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Impacts of the Future Steps 
Rural We l f a r e - t o ■ Wo r k Program: 
Key Findingsand Lessons 



' I ' he Rural Welfare-to-Work (WtW) Strategies Demonstration Evaluation is using 
I rigorous experimental designs to build knowledge about strategies to help low- 

X income famihes in rural areas strive toward sustained employment and self- 
sufficiency. This report examines the 18-month impacts of the Future Steps program 
on the employment, earnings, welfare dependence, and weU-being of work-ready low- 
income people in rural, southern Illinois. D uring a recent two-year period, this employment- 
focused case management program served mandatory TANF and food stamp recipients, as 
weU as low-income volunteers. Through job search and placement assistance, skill-building 
and support services, and postemployment assistance. Future Steps aimed to help chents 
overcome obstacles, develop skills, find and keep good jobs, and progress toward economic 
independence. The program was viewed as potentially effective since it (1) teamed the 
welfare agency with a community college, (2) benefited from the local connections of 
program staff, and (3) provided small caseloads of 15 to 35 active chents per case manager. 

Mathematica Pohcy Research, Inc. and its subcontractors. Decision Information 
Resources, Inc. and the Rural Policy Research Institute, are conducting this evaluation with 
funding from the U.S. Department of Health and Human Services, Administration for 
Children and Famihes. Based on a random assignment evaluation of the Future Steps 
program, this report addresses three key sets of research questions: 

1. How was Future Steps implemented and operated, and what did it cost? 

2. How effective was Future Steps at increasing employment and earnings, reducing 
welfare dependence, and improving other measures of weh-being? 

3. What are the imphcations and lessons for pohcy, programs, and evaluations? 



Key Findings 

Program Implementation and Costs 

• The core program model was implemented largely as intended. Many participants 
received services and support for an extended period— 16 months, on average. Future 
Steps offered more support than was otherwise available in the local area. 

• Significantly more program group than control group members received employment- 
related services and various types of supportive services. Moreover, the program group 
scored higher than the control group on self-efficacy and future orientation scales. 




• D espite a high average level of service use, more than one- fifth of all Future Steps chents 
received few services and had little contact with the program. Those who did not obtain a 
job or were served during the program's second year were more likely to be in this group. 

• One element of the program model— employer-focused job readiness and vocational 
training— was never implemented, partly because Future Steps did not invest adeguate 
time and effort into planning for it and building the necessary employer relationships. 

• The cost to operate Future Steps during a one-year, steady-state period was estimated as 
$333,214, which translated into an overall cost of $2,901 per participant. 



Impacts on Employment, Self-Sufficiency, and Well-Being 

• There was no evidence that Future Steps improved employment and earnings or reduced 
welfare dependence and poverty. At the 18-month followup, a httle more than half of 
sample members were employed, and close to two-thirds lived in poverty. 

• There was no evidence of program impacts for subgroups of clients defined by which year 
they participated or how employable they were when they enrolled. 



Possible Reasons for the Lack of Program Impacts 

• Some chents received few services or insufficient ones, which partly reflects the hmited 
training and support given to case managers, who varied in skihs and initiative. 

• Future Steps was not able to capitahze on the employer connections and job-training 
resources that its community college partner offered. This difficulty reduced the scope of 
the program's job readiness and job placement efforts. 

• Regardless of program implementation, the small and insignificant magnitude of the 
program versus control group differences in key outcomes suggests that basic case 
management is not hkely to be an adeguate intervention for improving employment and 
self-sufficiency in rural areas. Stih, it may be an important piece of a stronger intervention. 

Lessons FOR Future Programs in Rural Areas 

• In implementing programs in rural areas, local staff connections and initiative appear vital. 
In addition, adeguate compensation is important for staff recruitment, retention, and 
performance; careful training and ongoing support are essential when staff are in 
dispersed areas; and organizational performance incentives may have special value. 

• Cohaborative efforts between job developers and employers may be essential for strong 
interventions in rural areas. Economic development through wage subsidies, tax credits, 
and low-interest employer loans, and community improvements such as public van 
services, low-cost car loans, and good-guahty, accessible child care may also have value. 

• Evaluations of rural programs must address challenges related to constraints of scale. The 
sample size and power of an evaluation can be increased by expanding a program's 
enrohment period, target population, geographic catchment area, or number of locations. 
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Executive Summary 



The Rural Welfare- to- Work (WtW) Strategies Demonstration Evaluation is building 
knowledge about strategies to help low-income rural families strive toward sustained 
employment self-sufficiency, and improved family well-being. The demonstration arose in 
response to a lack of evidence about how to meet the particular needs of low-income people 
in rural areas. In rural labor markets, for instance, jobs are generally scarcer than in urban 
ones, and the available jobs more often involve low wages or part-time work. Education and 
training and other services may be more difficult to obtain. A lack of pubhc transportation 
can make access to jobs and services complicated. Moreover, tight-knit social networks can 
make jobs more difficult to obtain for long-term residents with poor personal or family 
reputations or for newcomers with few local ties. 

This report from the Rural WtW Strategies Demonstration Evaluation examines the 
impacts of the Euture Steps program on the employment, self-sufficiency, and well-being of 
work- ready low-income people in rural Illinois, including mandatory Temporaiy Assistance 
for Needy EamUies (TANE) and food stamp recipients, as well as low- income volunteers. 
This experimental design evaluation of Euture Steps provides the first rigorous test of a 
promising employment-focused case management model in a rural welfare-to-work context. 
Similar case management models have already been tested in urban areas, but demonstrated 
few positive effects. Euture Steps was an improvement over previous models, however, and 
was viewed as potentially effective, for several reasons. The model (1) responded to the 
scarcity of services and jobs by teaming the welfare agency with a regional community 
college; (2) drew on the local connections of program staff to help chents gain access to 
services, resources, and jobs; and (3) provided very small caseloads of between 15 and 35 
active clients per case manager. 

Mathematica Policy Research, Inc. (MPR) and its subcontractors, D ecision Information 
Resources, Inc. and the Rural Pohcy Research Institute, are conducting the Rural WtW 
Strategies Demonstration Evaluation with funding from the U.S. Department of Health and 
Human Services, Administration for Children and EamUies. In this report, we (1) chronicle 
the design and operation of Euture Steps and assess program participation, service dehveiy, 
and costs; (2) examine 18-month impacts on employment, earnings, welfare dependence, and 
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well-being; and (3) draw lessons and recommendations about implementing, designing, and 
testing future welfare- to-work programs in rural areas. 

The Rural Welfare-to-Work Strategies Demonstration Evaluation 

The Rural WtW Strategies Demonstration Evaluation is using rigorous experimental 
designs to assess whether two promising programs— Future Steps and Building Nebraska 
Families (BNF)— improve the employment, earnings, and well-being of low-income people 
in rural areas. While Future Steps served a broad segment of the low-income population, 
BNF (an innovative family life skills education program) targets hard-to-employ TANF 
recipients. Like Future Steps, BNF involves a partnership between a state welfare agency 
and a postsecondaiy educational institution (in this case, a university-based cooperative 
extension service). The evaluations of these two programs are on a staggered schedule, as 
enrollment was completed a year and a half earlier in Ilhnois than in Nebraska. This report 
focuses on 18-month Future Steps impacts; a forthcoming report in summer 2006 will focus 
on 18-month BNF impacts. Future reports wiU present 30-month impacts and cost-benefit 
findings for both programs. 

This evaluation of Future Steps addressed three sets of research guestions: 

1 . How was Future Steps implemented and operated, and what did it cost? 

2. How effective was Future Steps at increasing employment and earnings, 
reducing welfare dependence, and improving other measures of well-being? 

Was the program more effective for certain subgroups of clients? 

3. What are the imphcations and lessons for policy, programs, and evaluations? 

An experimental design, coupled with a multifaceted data analysis strategy, was used to 
examine the effectiveness of Future Steps and draw lessons from it. Using random 
assignment, 630 people ehgible for scarce program slots were assigned to either the 
treatment group or a control group (313 and 317, respectively). We estimated program 
impacts by comparing mean outcomes for the two groups 18 months after random 
assignment, and we improved the precision of the estimates by using multivariate regression 
methods. We obtained data to support the impact analyses from three primaiy sources: (1) 
a baseline information form completed by all sample members at the time of random 
assignment; (2) a comprehensive, 45-minute telephone interview administered 18 months 
after random assignment (with an overall response rate of 83 percent); and (3) Ilhnois state 
administrative records data for all sample members on employment, earnings, and pubhc 
assistance. Complementing the impact analyses, we conducted a thorough assessment of 
program implementation and costs. We rehed on three data sources: (1) service use data 
tracked through a customized program information system, (2) gualitative information from 
two rounds of in-depth site visits, and (3) insights from focus groups with both program and 
control group members. 
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Future Steps: Promoting Weleare-to-Work in Rural Illinois 

Through extended case management, Future Steps intended to help clients not only 
overcome obstacles and develop practical and vocational skills, but also find and keep good 
jobs and progress toward economic independence. The program design envisioned several 
core services or elements: (1) an assessment of skills and interests; (2) individualized job 
search, job placement, and skills enhancement plans; (3) help in overcoming personal and 
logistic^ barriers through outside referrals, $500 in flexible supportive service payments, and 
mentoring; and (4) extended postemployment support. In addition, employer-focused job 
readiness and vocational training was part of the original program plans. However, as 
described below, this program element was not reahzed. Figure 1 illustrates the key 
elements of Future Steps and how they aimed to support clients' efforts toward employment 
and self-sufficiency. Chents generally received regular services and support from the 
program until they had found a job and been employed for three months. After that, they 
accessed services as needed, which was weU over a year for most. 

Future Steps operated as a partnership between the job placement center at Shawnee 
Community College (SCC), based in Ulhn, Ilhnois, and the Illinois Department of Human 
Services (IDHS). The program operated from July 2001 to September 2003, serving more 
than 300 participants across five counties in an economically distressed region of southern 
Ilhnois. The Future Steps staff included a part-time program director (who doubled as the 
SCC placement center director), a program coordinator, and five case managers (cahed 
"career specialists"), who were stationed in each of the five county- level IDHS offices. Most 
Future Steps chents were referred to the program by ID HS staff. 



Assessment oe Program Implementation and Costs 

To put the impact findings into context and to identify operational issues that may be 
unigue to a rural context, it is important to understand the implementation of the program. 
Here, we highhght important successes and challenges in implementing Future Steps. 

• The core Future Steps program was implemented largely as intended, with many 
participants receiving substantial support over an extended period. 

Future Steps staff, by and large, provided the core services of the program successfuhy. 
Many chents received services at a substantial level of intensity over an extended period. 
Moreover, Future Steps appeared to offer substantiahy more individualized support to 
chents than was otherwise available in the local area. 0 n average, chents had 25 contacts 
with a Future Steps career speciahst over a 16-month period. These contacts often 
comprised several activities, such as employment assistance, support for transportation and 
other service needs, and mentoring on personal and family matters. The average client also 
received three or four supportive service payments, totaling nearly $300. 
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Figure 1. The Role of the Future Steps Program in Supporting Work and Economic 
Independence 
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• Despite a high level of sendee use for many, some Future Steps chents— 
particulaiiy those who did not obtain employment or those who were served 
during the evaluation 's second year— received few sendees. 

While about two-fifths of clients received substantial services over an extended period, 
one-fifth received few services and had httle contact with career speciahsts. Moreover, 
nearly 3 in 10 chents did not receive a supportive service payment. In general, clients who 
became employed during the follow-up period participated more actively and received more 
services than clients who did not become employed. Service use was also broader and more 
intensive among clients who enrobed during the demonstration's first year, when the 
program was better implemented and service delivery conformed more closely to the 
program model. During the second year of the evaluation period, the operation and service 
delivery of Future Steps weakened, largely because of staff turnover and budget pressures 
within see. 

• One element of the program model— employer-focused joh readiness and 
vocational training— did not materialize. 

Future Steps intended to enhance clients' work readiness and job-specific skhls by 
working with local employers to develop and implement customized job training. Future 
Steps planned to buhd on See's vocational- training resources and community connections 
to make it easier to develop collaborative job training efforts with employers. It was 
intended that employer-based training would help prepare clients for, and connect them to, 
good jobs in the local area— jobs that offer decent wages and a chance for benefits and 
advancement. This customize employer training, as envisioned, would have included work 
readiness training, hfe skihs birilding, and job-specific vocational instruction, all of which 
see could have helped develop and implement. Unfortunately, this program element was 
never My developed or implemented, for reasons described in greater detail below. 

• Although Future Steps appeared to improve chents' personal functioning, some 
chents expressed concerns about the program's ^ihty to provide effective 
assistance in a weak job market. 

Program group members scored significantly higher than control group members on 
scales that measured self-efficacy and orientation toward the future. Likewise, during the 
focus groups, many clients reported that Future Steps helped strengthen their motivation 
and self-confidence and gave them a sense of control over their lives. Even though Future 
Steps made many clients feel good about themselves and their efforts, some expressed 
concerns about the lack of readily available employment opportunities in the area and the 
limited services Future Steps provided to help address this. 
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• The average overall cost of serving a Future Steps participant was $2,901 

For the evaluation's cost study, we examined the market value of all resources used to 
operate the program during a one-year, steady-state period of operations. We estimated that 
the annual cost to operate Future Steps was $333,214, which translated into an overall cost 
per participant of $2,901. We estimated that Future Steps cost somewhat less per participant 
than comparable programs from the Evaluation of the Welfare-to-Work Grants Program 
(Perez-Johnson et al. 2002). 

Impacts ON Employment, Self-Sufficiency, and Well-Being 

Eighteen months after program group members enrolled in Future Steps, we 
hypothesized that they would have, on average, higher employment rates and earnings, lower 
TANF and family poverty rates, and greater levels of personal and family weU-being than 
control group members. For Future Steps to effect these improvements, the program would 
have to deliver more services to program group members than were received by control 
group members. 

• Significantly more program group than control group memhers received services. 

Using the 18-month follow-up survey data, we compared all the services and resources 
used by both program group and control group members during the foUow-up period. 
Program group members were significantly more likely than their control group counterparts 
to report having received various types of assistance (Figure 2). For example, nearly h^ of 
aU program group members reported on the 18-month follow-up survey that they had 
received some t^e of employment preparation assistance, with job search and job 
placement assistance the most common form. Program group members were much more 
likely than their control group counterparts to receive logistical support, including help 
finding and paying for transportation, child care, and job-related clothing, tools, or supplies. 
Higher fractions of program than control group members received life skills training and 
mediation services, though only about one-tenth of program group members received each 
of these services (not shown). 

• There is no evidence that Future Steps improved employment and earnings. 

Despite the significant service use differences. Future Steps did not translate into 
positive impacts on employment and earnings (Figure 3). Three-quarters of both groups 
were employed at some point during the follow-up period, and a httle more than half were 
employed 18 months after random assignment. Earnings also were similar for the two 
groups. Job turnover was common, and program group members held significantly more 
jobs than control group members. More than two-fifths of the program group, compared 
with one-third of the control group, held two or more jobs during the foUow-up period. 
The job changes the program group made, however, did not reflect a move into better or 
higher-paying jobs. Both program group and control group members held similar, low- wage 
jobs— only $6.78 per hour for program group members, on average (not shown). 
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Figure 2. Services Received During the 18-Month Foiiow-Up Period 

Percentage 



50 47*** 




Any Employment- Job Search/ Job Readiness Any Education Help Paying Help Paying Help Paying for 
Related Job Placement Training or Training for Child Care for Transportation Job-Related 

Assistance Assistance Clothing, Tools, 



or Supplies 



■ Program Group □ Control Group 



Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica 
Policy Research, Inc. 

Note: Estimates were adjusted using multivariate regression methods, and data were weighted to 

account for survey nonresponse. Standard errors of the estimates account for sample weights. 

*/**/•** Significantly different from zero at the .05/.025/.01 level, one-talled test. 



• Future Steps did not reduce welfare dependence or improve self-sufficiency. 

A year and a half after random assignment, there were no significant impacts on the 
receipt of TANF, food stamps, or other forms of public assistance (Figure 4). During the 
month before the 18-month follow-up survey, similar fractions of both groups reported 
receiving pubhc assistance: about 1 in 7 received TANF, and about 3 in 4 received food 
stamps. In addition, the program did not have an effect on household income or poverty. 
At the 18-month followup, more than two-thirds of program group members hved in 
households whose monthly income was below the federal poverty level (Figure 4). 
Hardships were also similarly pervasive for both groups. During the follow-up period, half 
of aU sample members faced three or more serious personal or logistical challenges— for 
example, challenges relating to health, transportation, child care, housing, and food (not 
shown). Such hardships can make it difficult to get a job, stay employed, and progress 
toward self-sufficiency. 
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Figure 3. Employment and Earnings for the 18-Month Follow-Up Period 
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■ Program Group □ Control Group 



Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica 
Policy Research, Inc. 

Note: Estimates were adjusted using multivariate regression methods, and data were weighted to 

account for survey nonresponse. Standard errors of the estimates account for sample weights. 

T*/*** Significantly different from zero at the .05/.025/.01 level, one-tailed test. 



• There was no evidence of program impacts for subgroups of clients defined by 
which year they participated or how employable they were when they emvUed. 

Future Steps was better implemented during its first year. Although the pattern of the 
findings shows hints of better program effects on economic outcomes for clients who 
enrolled and were served during this first year, the program-control differences were 
insignificant and very small. For other subgroups, differences in key outcomes were neither 
significant nor informative. In particular, there was no evidence that the program improved 
employment, earnings, or self-sufficiency for cherts who were either more employable or 
less employable when they enrohed. 

Possible Reasonsforthe Lackof Program Impacts 

Three factors might help explain why Future Steps did not have impacts on 
employment, earnings, and self-sufficiency. The first two relate to shortcomings in 
implementing program services. The third factor, and likely the more important, relates to 
inadeguacies in the scope of the program intervention, as designed. Each of the factors, 
described below, might have weakened the ability of Future Steps to effect an overall change 
in cherts' employment status and self-sufficiency. 
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Figure 4. Welfare Dependence and Poverty Status for the 18-Month Follow-Up Period 




Ever Received Received TANF Ever Received Received Food Living in Poverty 
TANF During in Month Before Food Stamps Stamps in at 18-Month 

18-Month 18-Month During 18-Month Month Before Foiiowup 

Foilowup Foiiowup Foiiowup 18-Month Foiiowup 



■ Program Group Flousehold □ Control Group Flousehoid 



Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey and administrative records 
data from the state of Illinois. The survey was conducted and the administration records data 
were compiled by Mathematica Policy Research, Inc. 

Note: Estimates were adjusted using multivariate regression methods, and data were weighted to 

account for survey nonresponse. Standard errors of the estimates account for sample weights. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-tailed test. 



• Some clients received few services or insufficient ones, which partly reflects the 
limited training and support given to career specialists, who varied in skills, 
initiative, and experience. 

Although the basic Future Steps case management model was reasonably well 
implemented, there were gaps in the program's coverage of clients. Services may not have 
gone far enough, even during the program's better-implemented first year, in connecting 
many clients with jobs or helping them overcome challenges. While some clients may not 
have wanted or needed services, staff may not have been able to reach others or help them 
in a meaningful way. Service use was particularly low among clients who were never 
employed during the follow-up period, suggesting that the program might have done more 
to help these clients secure employment. 

Overall, Future Steps did not prepare or support its career speciahsts at a level 
commensurate with the high expectations it placed on them. Although the career specialists 
were dedicated, caring, and hard-working professionals, they varied in skiU and initiative and 
some had limited education and prior experience. The program's incomplete coverage in 
serving clients partly reflects the limited training and support the career speciahsts received. 
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• Future Steps was not able to capitalize on the employer connections and job- 
training resources that its community coliege partner offered, thus lessening the 
scope of its job readiness and job placement efforts. 

A community college's education and training opportunities, job placement and career- 
planning resources, and employer connections can be beneficial to welfare-to-work efforts. 
As a locally respected institution, a community college can also encourage program 
enrollment and reduce the stigma often attached to welfare programs in rural areas. 

D espite the many advantages of working with a community college. Future Steps did 
not effectively capitahze on the employer connections and job-training resources the college 
offered. Because Future Steps did not develop and implement its employer-focused job 
training, the potential reach of the program's job readiness and job placement efforts was 
reduced. Future Steps did not invest adeguate time and effort into planning for this 
program component and building the employer relationships necessary to support it. Fully 
capitahzing on the college's connections and resources would have been easier had Future 
Steps designated more staff resources for it, such as a specialized job developer or a full-time 
program director, rather than a part-time one. In addition, if this component had been pilot- 
tested, as were the other components of the Future Steps model, it would have had a better 
chance of success. 

• Regardless of program implementation, basic case management is not likely to 
be an adequate intervention for improving employment and self-sufficiency in 
rural areas. Still, it may be an important piece of a stronger intervention. 

The evaluation of Future Steps provided a good test of the effectiveness of basic 
employment-focused case management in a rural setting. The absence of program impacts 
on key outcomes suggests that case managers, no matter how skilled and effective, are 
limited in what they can help their chents accomplish. If, as in southern Ilhnois, an area 
lacks good jobs and adeguate services hke child care and transportation, then case 
management services alone appear unlikely to overcome such hmitations. Indeed, the 
findings indicate that control group members, without the assistance of Future Steps, were 
just as likely as program group members to secure and maintain the mostly low- and 
semiskiUed jobs that were available. Moreover, given the small and insignificant magnitude 
of the program versus control group differences in key outcomes, any gains from a better- 
implemented case management intervention would veiy possibly still have led only to 
marginal improvements in the economic prospects of the relatively work-ready population 
Future Steps served. 

Overall, the findings suggest that basic case management services are not likely to be an 
adeguate intervention to help low-income people in distressed rural areas address obstacles, 
find lasting employment, and become self-sufficient. This conclusion supports what other 
studies have already suggested about employment-focused case management in urban areas 
(Rangarajan and Novak 1999). The findings appear to suggest a need for additional program 
elements to help low-income people in rural areas move to sustained employment and self- 
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sufficiency. Community disadvantages like high unemployment and limited job 
opportunities may be difficult to overcome without expanded services and community and 
economic development efforts that go beyond case management. Case management, though 
not a program solution on its own, may still be a useful way to provide beneficial services to 
clients, as long as services are coupled with other efforts that address individual and 
community employment challenges in a more systematic and substantial way. 



Lessons FOR Future Welfare -to-Work Programs in Rural Areas 

The evaluation findings demonstrate the inherent challenges in helping low-income 
people in rural areas make the transition to employment and self-sufficiency. Despite the 
absence of impacts on employment, earnings, and self-sufficiency, this Rural WtW 
evaluation of Future Steps still offers useful lessons for shaping the implementation, design, 
and evaluation of future programs in rural areas. These lessons may help identify promising 
program models for future demonstrations and provide guidance for implementing and 
testing them successfully. 



Implementing Programs in Rural Areas 

To implement strong programs in rural areas, administrators must adapt to conditions 
specific to rural places: few good jobs, limited services and resources, and the geographic 
dispersion of people and places. In addition, tight-knit local communities can impede 
employment efforts for those with a poor reputation or few ties to a local area. These 
conditions should be considered in the design and implementation of program models in 
rural areas. The experiences from the Future Steps evaluation suggest several lessons on 
program operation and case management service delivery in a rural context. Although the 
potential effect of these lessons or factors is not known, each may be important for 
implementing programs in rural areas successfully. 

• L ocal staff connections and initiative appear to be important elements of 
successful sendee delivery in rural areas. The most capable case managers in 
rural areas may be those famihar with their communities and able to identify, 
and connect clients to, opportunities and services. Staff connections and the 
initiative and resourcefulness to use them can help in making referrals, 
identifying job openings, vouching for chents to prospective employers, and 
mediating chents' problems. Vouching for clients may have special value in 
rural communities, in which a poor personal or family reputation can negatively 
affect a person's economic prospects. 

• To promote staff recruitment, retention, and a high degree of skill and 
performance, an adequate investment in staff compensation is important 

The guahty of case management service dehveiy depends greatly on staff 
capabilities. Program leaders in rural areas may find it chahenging to recruit and 
keep staff who have the necessary combination of skihs, community famihaiity, 
and professionahsm. Offering fuh-time positions with a fuh set of employment 
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benefits and a competitive wage will likely help in recruiting and keeping 
gualified local staff for programs in rural areas. 

• Careful training, oversight, and ongoing support are essential for staff in 
dispersed, rural areas. Staff in dispersed, rural areas are often expected to 
exercise substantial independence and discretion in their daily work. T o work 
effectively in this context reguires a high level of maturity, professionalism, and 
self-motivation, as well as broad skills. Training, oversight, and ongoing support 
are important for guiding the work of staff in dispersed locations. While this is 
true for dispersed staff in general, it is especially important for those who may 
have limited education and professional experience. 

• Incorporating performance incentives into agreements with partner 
organizations may help programs stay focused on goals. Performance- 
based incentives can encourage program staff to meet predetermined objectives, 
such as those related to program enrollment, job placement, job retention, and 
the use of supportive service funds. This t^e of management tool may be 
particularly useful when programs will not be sustained at the end of a funding 
period, or when partners with diverse organizational missions are hkely to face 
resource and staffing constraints. 



Designing Welfare-to-Woik Programs in Rural Areas 

One of the fundamental goals of the Rural WtW demonstration is to identify program 
strategies that pohcymakers should consider in designing future welfare-to-work programs. 
The absence of impacts in this evaluation of Future Steps imphes a need for stronger 
interventions that target low-income workers in rural areas. We highhght several program 
strategies below that may help strengthen future welfare-to-work interventions in rural areas. 
Interventions that include one or more of these strategies may be good candidates for 
further evaluation, as their potential effectiveness is uncertain. 

• Building Unkages with employers to promote joh opportunities may take 
on added importance in rural areas with few good jobs. Involving job 
developers in program efforts may be essential. By working collaboratively with 
employers, programs may help identify and develop good job opportunities for 
their chents and may help prepare and train clients for jobs. At a minimum, 
creating employer hnkages reguires substantial planning and effort. Employers 
must perceive benefits in worMng with a welfare or workforce agency and must 
overcome concerns about prospective employees' work attitudes and 
dependability. D eveloping relationships with employers and job opportunities 
for chents may reguire investing in specialized job development services. 
Welfare and workforce agencies and their programs may have more success 
cohaborating with employers if they hire or contract with professional job 
developers. 
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• Economic development represents an important strategy for improving 
the employment prospects of low-income workers in distressed rural 
areas. Kurd areas with few job opportunities may need to look beyond welfare- 
to-work interventions. For example, wage subsidies, tax credits, and low- 
interest loans to employers are incentives that state welfare and workforce 
agencies and other policymakers might consider for disadvantaged rural areas. 
Such tools can act as incentives for employers to expand their business, create 
new jobs, hire low- and semiskiUed workers, and offer services like on-site child 
care and van shuttles. In addition, community development efforts may help 
improve the economic prospects of low-income workers in distressed areas by 
alleviating challenges they face related to housing and other services. 

• Systematic improvements to the availability of logistical services like 
child care and transportation may be needed in many rural areas. Reliable 
transportation and accessible, good-guahty child care are support services 
essential to labor market success. Public van services, low-cost car loans, and 
good-guahty, accessible care during nonstandard work hours may be particularly 
vital in rural areas with hmited resources. 



Evaluating Programs in Rural Areas 

Conducting evaluations in rural areas is inherently chahenging given constraints of scale. 
Resources and population density are both smaller in rural areas than in urban ones. Two 
factors may be useful for evaluators and funders to consider in planning future program 
evaluations in rural areas. 

• Evaluators of rural programs can increase the sample size and power of 
an evaluation by expanding a program's enrollment period, target 
population, catchment area, or number of sites. Low population densities 
in rural areas can make it difficult to form treatment and control groups large 
enough to detect impacts. As in this evaluation, evaluators of new, promising 
rural programs may need to think creatively about ways to increase a program's 
scale and an evaluation's sample size. This can be done, for example, by 
extending the enrollment period, expanding the target population, increasing the 
geographic catchment area, or adding program locations or sites. To broaden 
the number of small-scale programs that can be considered for future 
evaluations, it might be particularly useful to implement a program model in 
additional sites (for example, in neighboring counties or other parts of the same 
state). This may be the most effective way to substantially increase the reach of 
a small program and make it large enough for an evaluation. If additional sites 
are added, however, more central guidance and oversight will likely be needed to 
ensure that sites consistently implement the same program model. 
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• To strengthen program models and their implementation, future 
evaluations in rural areas might benefit from more intensive technical 
assistance before and during a demonstration. Because of limited local 
resources, technical assistance may take on added importance in rural areas. 
Rural areas may generally have fewer resources to invest in the development of 
new and innovative program models. Moreover, when rural programs 
encounter operational challenges, it can be relatively difficult to recover and 
adapt guickly because of fewer local resource choices and options. By design, 
active technical assistance before and throughout a demonstration evaluation 
might help strengthen a program model and its implementation. 
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Chapter I 
Introduction 



K ural low-income families trying to achieve economic independence face distinct 
challenges. Low-income people in both rural and urban places may encounter such 
barriers to employment as limited job skills and inadeguate child care. However, the 
economic and geographic conditions in rural areas often create additional hurdles for welfare 
recipients and other disadvantaged people hoping to find jobs, maintain employment, and 
secure long-term well-being. In rurd labor markets, jobs are generally scarcer than in urban 
ones, and the available jobs more often involve low wages or part-time work. Education and 
training opportunities, as well as services such as health care and mental health care, also 
may be more difficult to obtain. In addition, a lack of public transportation, common in 
rur^ areas, can make access to existing jobs and services difficult. Moreover, tight-knit 
social networks can further hamper employment efforts. A poor personal or family 
reputation can negatively affect someone's economic prospects by making local employers 
less willing to hire that person (Findeis et al. 2001). On the other hand, for people with few 
ties to an area, a lack of local connections can make jobs more difficult to obtain. 

The work reguirements in the Temporaiy Assistance for Needy Families (TANF) and 
Food Stamp programs have provided a strong impetus for low-income people to find and 
keep jobs and for states and locahties to design and operate programs that help people 
achieve labor market success and economic independence. The TANF program, established 
under the Personal Responsibihty and Work Opportunity Reconcihation Act (PRWORA) of 
1996, imposes work reguirements and a five-year hfetime hmit on cash assistance for most 
recipients. Through PRWORA, the Food Stamp Program similarly instituted employment 
and training reguirements for many recipients, and, in some cases, benefit limits. These 
reguirements and hmits to TANF and food stamp benefits have motivated many low- 
income people to prepare for and find jobs. 

The Rural Welfare- to- Work (WtW) Strategies Demonstration Evaluation is rigorously 
evaluating the effectiveness of programs in rural areas that were designed to address the 
unigue challenges facing the rurd poor as they strive toward sustained employment and self- 
sufficiency. This evaluation arose in response to a lack of evidence about how to help low- 
income workers in rural areas progress toward economic independence. Using rigorous 
random assignment designs, the evaluation is assessing whether two programs— Ilhnois 
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Future Steps and Building Nebraska Families (BN F)— improve the employment, earnings, 
and well-being of low-income people in rural areas (see text box). Future Steps is an 
employment-focused case management program that targets TANF and food stamp 
recipients and other low-income volunteers, while BNF is a family hfe sldlls education 
program that targets hard- to-employ TANF recipients. Both represent partnerships between 
welfare agencies and postsecondaiy educational institutions (a community college in Illinois 
and a university-based cooperative extension service in Nebraska). Mathematica Policy 
Research, Inc. (MPR) and its subcontractors, D ecision Information Resources, Inc. and the 
Rural Pohcy Research Institute, are conducting the evaluation with funding from the U.S. 
Department of Health and Human Services, Administration for Children and Families 
(ACF). 

This report provides a comprehensive analysis of Ilhnois Future Steps, a promising case 
management program. It combines 18-month impact analysis findings with assessments of 
program implementation, participation, and costs. In this chapter, we describe the 
demonstration's site selection process, key features of Future Steps, and the evaluation 
design and methods. In Chapter II, we chronicle the Future Steps experience, describe the 
target population and community context, discuss findings related to program operations 
and participation, and examine program costs. In Chapter III, we present impact findings, 
including treatment versus control group differences in service use; impacts on employment, 
earnings, welfare dependence, and guahty of hfe; and findings for key subgroups. In Chapter 
IV, we offer lessons and recommendations related to designing, implementing, and 
evaluating welfare- to-work programs in rural areas. 



Selecting Promising Rural Programs eor the Demonstration 

Through the Rural WtW Strategies Demonstration Evaluation, pohcymakers wanted to 
learn more about rural welfare and workforce program approaches. During the evaluation's 
early phases, MPR worked closely with ACF to identify promising local programs in several 
states that could be evaluated using rigorous, random assignment methods. 



Program Selection Criteria 

In selecting programs, we focused on existing programs in rural locations that were 
designed to address chahenges unigue to rural areas. The rural programs were reguired to 
meet several important criteria: 

• Local programs are strong. Programs were selected only if they were viewed 
as innovative, weh run, and important from a national pohcy perspective. In 
addition, program services needed to be substantially different from the local 
services otherwise available to ehgible chents so that meaningful differences in 
service receipt and impacts could be reasonably achieved and measured. 
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The Rural Welfare-to-Work Strategies Demonstration Evaluation 

The Rural Welfare-to-Work Strategies Demonstration Evaluation is the first 
rigorous, systematic evaluation of programs designed to help move rural famihes from 
welfare to sustained employment, job progression, and economic independence. The 
evaluation includes three complementaiy studies: 



• Implementation and Cost Study. The evaluation includes an in-depth 
examination of the context, operation, and costs of both the Illinois Future 
Steps and the BNF programs based on site visits, program records, and 
chent focus groups. We are identifying important implementation issues, 
examining how programs achieved observed results, drawing lessons about 
service dehveiy challenges and innovation, and developing estimates of 
program costs. 

• Impact Study. A random assignment design is allowing evaluators to 
determine what difference the programs make in clients' employment, 
earnings, welfare dependence, and well-being. For both Future Steps and 
BNF, people eligible for scarce program slots were assigned to either a 
treatment group (who were offered program services) or a control group 
(who were not offered program services but were able to use all other 
available services). We are comparing the behaviors and outcomes of these 
two groups over time to determine each program's net impact. 

• Cost-Benefit Study. The evaluation will calculate estimates of net 
program cost-effectiveness based on data from the impact and 
implementation studies and published research (assuming that impacts are 
positive and significant). Study plans include analyses of the distribution of 
costs and benefits from several perspectives— participant, government, and 
society at large. 

The Rural Welfare-to-Work Strategies Demonstration Evaluation began in 2000 
and wiU extend through fall 2007. The two programs and evaluations were 
implemented on a staggered schedule, with random assignment and data collection 
completed a year and a half later in Nebraska than in Illinois. A report on cross-site 
program implementation lessons was finalized in early 2004. This report presents 18- 
month impact findings for Illinois Future Steps, along with an assessment of program 
implementation, participation, and costs. In summer 2006, we will present a similar set 
of findings for BNF. After that, we wiU report on 30-month impacts and cost-benefit 
findings for both programs. 
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• Random assignment is feasible. Because it was important to conduct a 
rigorous evaluation, a random assignment design was introduced. Random 
assignment was judged to be feasible if (1) the demonstration would not deny 
people access to services or benefits to which they were entitled, and (2) there 
were more people ehgible and suitable for the program than the program was 
funded to serve. 

• Potential research sample is adequate. The larger the sample size, the better 
the chance of rehably detecting program impacts. As a target for a balanced 
evaluation design, we established a minimum sample of 600 people (300 each in 
the treatment and control groups) enrolled over an 18- to 24-month period. 
This target can be difficult to achieve in rural areas, because of the small scale of 
the areas and population dispersion. 



Using a multistage process, we and ACF examined and considered 25 initiatives across 
20 states. Given the relatively small scale of many rural areas and programs, it was 
challenging to identify programs that met the selection criteria. Ultimately, we selected three 
distinct program models to study: (1) Illinois Future Steps, (2) Building Nebraska Famihes, 
and (3) Tennessee First Wheels (an interest-free car loan program). Unfortunately, the 
Tennessee First Wheels Program formally withdrew from the evaluation in September 2003, 
after the random assignment process was under way.^ To form sufficiently large samples, 
participating states broadened their target populations to include food stamp recipients and 
low-income volunteers (in Illinois) and additional rural counties (in Nebraska). 



T esting Case Management in a Rural Context 

The evaluation of Future Steps provides the first rigorous test of an intensive, case 
management model in a rural welfare-to-work context. By working with clients over an 
extended period. Future Steps was designed to prepare low-income people in rural areas for 
work and help them find and keep good jobs and make progress toward economic 
independence. Similar employment-focused case management models had already been 
tested in urban welfare-to-work contexts but demonstrated few positive effects. In 
particular, the Postemployment Services Demonstration (PESD) found few impacts on 
employment, earnings, and welfare dependence for the four urban case management 
programs it evaluated (Rangarajan and Novak 1999). Like Future Steps, these programs 
focused on case management services, without engaging in more systematic efforts to build 
clients' skills, cultivate employment opportunities, or expand local services. 



1 Tennessee withdrew due to difficulties enrolling clients into the program and shifting state budget 
priorities that reduced the program's funding. 
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Despite uncertainty about the effectiveness of welfare-to-work programs that 
emphasize case management an enhanced case management model like Future Steps was 
viewed as potentially effective in a rural setting, for several reasons. First by teaming with a 
regional community college, the state welfare agency positioned itself to respond to the 
scarcity of job opportunities and services in southern Illinois. D rawing on the resources of 
the community college. Future Steps aimed to prepare chents for work, help them find and 
keep good jobs, and build job training and development efforts with local-area employers. 
In so doing. Future Steps was intended to add value to the limited opportunities and services 
already present in the region. 

Second, Future Steps hoped to capitalize on existing community connections by hiring 
staff with strong local knowledge, familiarity, and ties. The program hoped that staff would 
act resourcefully in drawing on their local connections and knowledge to help chents gain 
access to services, resources, and employers; vouch for them as candidates for available jobs; 
and, as appropriate, mediate problems they faced. 

Finahy, because Future Steps case managers carried very smah caseloads, they were able 
to work with chents intensively. Caseloads averaged 15 to 35 active chents, plus 10 to 15 
transitional chents per case manager. This size was much lower than most other welfare- 
related case management programs, allowing for frequent, individualized mentoring and 
support both before and after chents became employed. For example. Future Steps 
caseloads were at least half the size of those in PE SD , which were 100 to 170 chents per case 
manager, on average (Rangarajan and Novak 1999). 



Welfare -TO-W oRK IN Rural Illinois: The Future Steps Program 

Future Steps encompassed a mix of employment-related services, including intensive 
job search and placement assistance; individuahzed supportive services, referrals, and 
mentoring; $500 in flexible supportive service payments; and extended postemployment 
support. Chents generally received regular support until they had found a job and been 
employed for three months. After that, they accessed Future keps services less intensively, 
as needed. The average length of enrollment was nearly 16 months. The program served 
more than 300 participants across five southern Ihinois counties from July 2001 to 
September 2003. A program diagram illustrates the key elements of the program and how 
they are intended to help chents enhance vocational and hfe skihs and address personal and 
logistic challenges as they prepare for work, obtain and maintain employment, and progress 
toward self-sufficiency (Figure I.l). 

Future Steps operated as a partnership between the job placement center at Shawnee 
Community Cohege (SCC), based in Ulhn, Ilhnois, and the Ilhnois Department of Human 
Services (IDHS). SCC staff ran Future Steps under a contract mth the IDHS. The 
program's staff included a part-time program director (who doubled as the SCC placement 
center director), a program coordinator, and five case managers (called "career speciahsts"), 
who were stationed in each of the five county- level ID HS offices. 
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Figure 1.1 . The Role of the Future Steps Program in Supporting Work and Economic 
Independence 
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Future Steps targeted both mandatoiy and voluntaiy participants. The program enrolled 
both TANF and food stamp clients who were required to work, as weU as other low- income 
people who volunteered.^ To be eligible, individuals had to be available and wUhng to work 
at least 30 hours per week. Most were referred to Future Steps by their IDHS caseworker, 
although some were referred by SCC or other organizations in the community. We estimate 
that between one-quarter and one-third of eligible Future Steps apphcants were volunteers.^ 
For mandatoiy chents, TANF and food stamp benefits could be sanctioned if clients failed 
to cooperate with Future Steps requirements. In practice, over the course of the 
demonstration. Future Steps recommended to IDHS that about one- tenth of clients be 
sanctioned, typically because the client had repeatedly faked to communicate or participate in 
meetings with their Future Steps case manager. 

The Future Steps service area covered five rural counties in the southern tip of Ilhnois 
(Figure 1.2). Among the combined population of 64,000, most residents in the area are 
white, with a substantial minority— about 13 percent— African American. Ak five counties 
offer very limited pubhc transportation and face economic chakenges, including poverty 
rates above the statewide average (as high as 24 percent in one county). Entry- level jobs are 
most commonly with retailers, fast-food restaurants, and nursing homes. The area has a few 
larger employers, including chemical and cement factories, state prisons, and a riverboat 
casino. The Future Steps program and its clients and local context are described in greater 
detail in Chapter II. 



Evaluation Design and Methods 

The goals of the Rural WtW Strategies Demonstration Evaluation are to evaluate the 
effectiveness of promising welfare-to-work interventions in rural areas, provide 
recommendations for improving programs, and guide future policymaking and program 
development in rural areas. In particular, this evaluation of Future Steps aimed to answer 
three key research questions: 



1. How was Future Steps implemented and operated, and what did it cost? 

2. How effective was Future Steps at increasing employment and earnings, 
reducing welfare dependence, and improving other measures of wek-being? 
Was the program more effective for certain subgroups of clients? 

3. What are the imphcations and lessons for policy, programs, and evaluations? 



2 Clients were considered low-income if they lived in households with income under 200 percent of the 
federal poverty level. 

3 It is not possible to identify more precisely the mandatory and voluntary participants, primarily because 
of changes in the food stamp participation requirements in southern Illinois midway through the 
demonstration. 
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Figure 1.2. Geographical Representation of the State of Illinois and the Five 
Future Steps Counties 
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Here, we describe the evaluation methods and data collection sources used to answer these 
questions. 

Implementation and Cost Study 

The implementation and cost study seeks to document the Future Steps program model 
and service dehveiy strategies, describe client experiences, assess program implementation, 
and, more generally, provide a context for interpreting the impact study findings. T o explore 
these topics, we relied on both quantitative data from the Future Steps Information System 
(FSIS) and qualitative information collected through in-depth program site visits. In 
addition, a careful methodological approach was used to develop an estimate of the cost of 
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the program. These data sources and methods together provided a detailed picture of the 
management and operation of the program. 

Service Use Data. Information on clients' program participation and service use 
comes from the FSIS, which we developed and maintained in collaboration with the 
program. The FSIS accommodates record-keeping tasks that staff members regularly 
perform, such as documenting the topics discussed in client meetings and the services 
provided to clients. The system also provides data on how frequently clients meet with staff, 
the intensity of staff/ client interactions, the type and quantity of supportive service 
payments made, and basic indicators of employment experiences. 

Site Visits and Focus Groups. A team of two researchers made two site visits to 
Future Steps, one in each of the two years of the demonstration period. These visits covered 
the entire five-county area and lasted four or five days each. Site visits included in-depth, 
semistructured interviews with staff from Future Steps, IDHS, and other local agencies; case 
reviews; and observations of program activities. We also conducted focus groups with 
program participants and members of the evaluation's control group to gather information 
on participants' program experiences, obstacles to employment, awareness of service 
availability, and perceptions of the program's helpfulness. 

Cost Study Methods. We built up an estimate of the aggregate cost of operating 
Future Steps during a one-year, steady-state period by using information obtained through 
in-depth staff interviews during the site visits, along with program expenditure records. In 
developing the cost estimate, we followed the methodological approach articulated by 
Thompson (1998), which has been used successfully in other social service program settings 
(Perez-Johnson et al. 2002; Ohls and Rosenberg 1999). We measured the market value of aU 
resources used to deliver services and operate the program during the cost period, including 
"off-budget" expenses that were donated, shared with other programs, or absorbed by an 
organization's general administrative structure. We also standardized our aggregate cost 
estimate by converting it into an average cost per participant. Chapter II provides additional 
detail on cost study methods. 



Impact Study 

We used an experimental design to determine the difference Future Steps made in 
employment rates, earnings, welfare receipt, and well-being. Using random assignment, 630 
people eligible for scarce program slots were assigned to either the treatment group or a 
control group during an 18-month enrollment period. A balanced design was used, with the 
probability of selection to the treatment group essentially equal to 50 percent (313 were 
assigned to the treatment group and 317 to the control group). Treatment (or program) 
group members were enrolled into Future Steps and offered program services (generally on 
the same day or within a day or two of random assignment), while control group members 
were not offered program services (although they had full access to aU other available 
services). Participation in the program was nearly universal; 93 percent of program group 
members participated in or received at least one activity or service, and all but one had some 
type of verbal contact with program staff. 
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The random assignment process was implemented correctly. The basehne characteristics 
of the two groups were very similar, and there were no systematic, significant differences 
between them. (Chapter II displays these characteristics.) In addition, careful monitoring of 
program enrollment throughout the demonstration ensured that no members of the control 
group enrolled in the program. 

Data Sources and Methods. We relied on three methods and sources to collect data 
for the study of program impacts: 



1. Baseline Infonnation Form. We collected baseline demographic and 
socioeconomic data on aU sample members just before random assignment 
using information forms developed for the evaluation and completed by sample 
members. 

2. Follow-Up Survey 18 Months After Random Assignment W e conducted 
a 45-minute follow-up telephone survey with sample members 18 months after 
they were randomly assigned. We achieved a response rate of 83 percent (520 
completes out of 630 sample members) using telephone interviewing methods, 
along with intensive field followup. The data collected from the survey were 
weighted to account for nonresponse. Appendix A contains a full discussion of 
survey data collection and weighting methods. 

3. Administrative Records from the State of Illinois. We obtained state-level 
administrative records data on aU sample members for reported quarterly 
employment and earnings and monthly TANF and food stamp receipt. Data 
were obtained for the full 18-month period after random assignment. 



Outcome Measures. The analysis assessed the effects of Future Steps on outcomes 
related to labor market success, dependence on pubhc assistance, family weU-being and 
poverty status, and individual and family functioning. The primaiy data source for the 
impact analysis was the 18-month survey, which provided a more detailed set of data than 
the administrative records, including variables on the characteristics of jobs held, income 
sources, and family income. We measured outcomes at specific points in time, as well as 
continuously. Point-in- time measures included such items as job characteristics, income, and 
hving arrangements, while continuous measures included such items as the duration of 
employment and welfare receipt. Depending on the data source, period-specific measures 
were defined for each month or quarter, as weU as for aggregated periods (such as the fuU 
18-month foUow-up period). 



4 Evaluation plans also include a 30-month follow-up survey. We plan to report on findings based on the 
Illinois 30-month survey data in fall 2006. 
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Analytic Methods. Since random assignment was used to create the program and 
control groups, we can attribute the subsequent differences in the groups' outcomes to the 
incremental services the program offered. We estimated impacts by comparing mean 
outcomes for the program and control groups 18 months after random assignment. The 
differences between the mean outcomes represent unbiased estimates of the average effects 
of Future Steps. 

To improve the precision of the impact estimates, we used multivariate regression 
methods. We controlled for relevant demographic and socioeconomic variables collected at 
baseline, as well as two key contextual variables (chents' county of residence and year of 
program enrollment) that we identified through the implementation study as factors that 
may have influenced the key outcomes.^ We estimate that the variance of the impact 
estimates was reduced by 15 percent as a result of using multivariate modehng. Using similar 
methods, we also examined program impacts for key subgroups. Basehne characteristics 
were used to define key subgroups. 

We identified program impacts if treatment group outcomes differed from control 
group outcomes by a margin that was statistically significant using a one-tailed test at the 95 
percent confidence level. Power calculations indicated that, to detect significant impacts 
using our survey sample, we needed to observe monthly earnings differences of about $100, 
TANF benefit differences of $35, and employment and welfare impacts of 8 to 10 
percentage points.® If the program had effects of these magnitudes, we had an 80 percent 
chance of detecting them. 



^ Chapter II, Table II. 1, shows the baseline demographic and socioeconomic characteristics that were 
used as control variables in our regression- adjusted models. The use of multivariate regression models adjusts 
for any random residual differences in the observable baseline characteristics of the program and control group 
members. We generally used ordinary least squares estimation techniques for continuous variables and logistic 
regression methods for binary outcomes. We also examined the sensitivity of the impact estimates and found 
that they were largely insensitive to alternative model specifications. For example, we ran Tobit regression 
models for key continuous outcome variables (such as earnings and TANF amounts) and found that the results 
were similar to the results found using ordinary least squares estimation techniques. 

6 Minimum detectable differences were somewhat smaller when administrative records data were used, 
since administrative data for aU 630 sample members were available. The evaluation was able to detect 
quarterly earnings differences of $275, monthly TANF benefit differences of $31, and employment and welfare 
impacts of seven to nine percentage points based on the administrative data. 
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Chapter II 

Implementing Future Steps 



' I ' 0 put the impact findings into context and to identify operational challenges that may 
I be unique to a rural context, it is important to understand the implementation of 
i Future Steps. Our implementation study suggests that Future Steps was 
implemented largely, but not entirely, as planned. Many chents received services at a 
substantial level of intensity and over an extended period, though some had relatively little 
contact with career specialists and received few services. During the demonstration's second 
year, the implementation of Future Steps weakened somewhat because of staff turnover and 
budget pressures in the organizations operating the program. 

In studying the implementation of Future Steps, we focused on several issues: (1) how 
Future Steps was developed and how closely its operation conformed to its program model; 
(2) the program's setting, in terms of economic conditions, service avahabihty, and chent 
characteristics; (3) clients' experiences as they participated in the program; and (4) the total 
costs of program operation. In this chapter, we first describe the development, service 
delivery, and target population of Future Steps, as weU as the local economic and policy 
environment. We then highlight key findings regarding program operation, participant 
experiences, and program costs. 



Program Development, Staffing, and Service Delivery 

Future Steps aimed to provide focused, personalized assistance to promote employment 
for low-income clients. The Future Steps program model featured services intended to 
identify clients' skills, remove barriers to work, match chents to job opportunities, and 
support chents after they became employed. In this section, we describe the implementation 
of this model, including its development, staffing, and service dehveiy process. (See text box 
for a summary of the key program elements.) 
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Key Features OF THE Future Steps Program 

• Program Model. Employment-focused case management, including job 
search and job placement assistance, referrals for services and training, up to 
$500 in flexible supportive service payments, mentoring, and postemployment 
support 

• Dumtion of Program Enrollment Average length of enrollment nearly 16 
months. Chents generally received a high level of support until they found a job 
and were employed for three months. After that, they accessed services as 
needed. 

• Target Population. Mandatory TANF and food stamp recipients, as well as 
low-income volunteers. An estimated one-guarter to one-third were volunteers. 
AH ehgible apphcants had to be available and wUhng to work at least 30 hours 
per week. For mandatory chents, TANF and food stamp benefits could be 
sanctioned if chents did not cooperate with the program. 

• Partner Organizations and Service Area. Ilhnois Department of Human 
Services and Shawnee Community Cohege operated the program in five rural 
counties in the southern tip of Ilhnois. 

• StafGng. Shawnee Community Cohege placement center director, program 
coordinator, and three to five career speciahsts. Career speciahsts were co- 
located in local pubhc assistance offices. 

• Caseload Size. 15 to 35 active chents, plus approximately 10 to 15 transitional 
chents per career speciahst. 



• Future Steps developed and operated through a productive coUahomtion between 
a state welfare agency— the fUinois Department of Human Services (fDHS), and 
an educational institution— Shawnee Community College (SCC). 

Future Steps emerged from an existing partnership between SCC and IDHS. The two 
organizations had cohaborated previously on Advancing Opportunities, an employment 
program operating at community coheges throughout Ilhnois. Advancing Opportunities 
aimed to prepare participants to enter the workforce through training and supportive 
services. In designing Future Steps, SCC and IDHS modified the Advancing Opportunities 
model to reflect lessons learned from that program and make the new program suitable for 
other sources of support, such as federal Welfare- to- Work funding. Compared to 
Advancing Opportunities, Future Steps placed more emphasis on rapid employment and 
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served a broader range of clients, including low-income people who were not current or 
former recipients of Temporary Assistance for Needy Families (TANF). Future Steps also 
established a more modest upper limit for supportive service payments ($500, rather than 
$1,200) but provided greater flexibility in how these funds were used and allowed staff 
discretion to exceed the $500 "soft cap" on payments to individual clients, as appropriate. 
According to Future Steps staff, these changes helped address the varied needs of their 
clients. 

The see placement center, which provides job search resources to students and the 
wider community, operated Future Steps under contract to IDHS. In its day-to-day 
operation. Future Steps reguired ongoing coordination between See staff and regional and 
local IDHS staff. Future Steps staff providing direct services worked in IDHS county 
offices to facilitate their access to cheats and IDHS staff. IDHS caseworkers identified 
clients suitable for Future Steps and worked with MPR to conduct random assignment for 
the evaluation. Future Steps staff regularly updated IDHS caseworkers and local office 
administrators on the employment and participation status of chents in the program. 
Occasionahy, staff from Future Steps and IDHS coordinated the dehvery of services for 
clients with particularly chahenging problems. 

• Program staff included several career specialists, who provided direct services, a 
part-time program director, and a hill-time program coordinator. 

Future Steps career specialists (case managers) were based in IDHS offices in each of 
the five program counties and provided direct services to Future Steps clients. Most career 
speciahsts were part-time employees. However, the specialist working in Alexander County, 
who carried a larger caseload than the others, worked fuh-time. According to program 
administrators, key guahfications for the position included a basic understanding of IDHS 
services, strong interpersonal and communication skihs, familiarity with the communities in 
which they worked, and the attention to detail needed for such tasks as completing case 
notes. Experience with IDHS programs was also considered helpful, and some career 
speciahsts were themselves former recipients of cash assistance. A college education was not 
a reguirement for the position, but some speciahsts did have bachelor's degrees, and others 
had taken college courses. 

Workers considered to be particularly strong performers were resourceful, self- directed, 
highly organized, and assertive in deahng with chents and IDHS administrators. In general, 
the more competent Future Steps workers appeared to be those who could act as advocates 
for their chents with employers or IDHS, while holding chents accountable for working 
toward their goals. 

Future Steps management staff initiahy consisted of the director of the SCC placement 
center and a program coordinator. (Staff turnover during the evaluation period led to 
changes in the administrative structure, as described in the section on Future Steps 
implementation.) The placement center director, who had been involved in designing 
Future Steps, provided overah leadership for the program, dedicating more than half of her 
time to Future Steps. Future Steps seemed to benefit in its early phases from her clear vision 
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and community connections. The director and the program coordinator shared 
responsibility for estabhshing pohcy and procedures for Future Steps and for supervising 
other program staff members. The coordinator also was a central point of contact for the 
career specialists, answering questions from staff in the field by email and telephone. 

Initial training for career specialists consisted of a daylong session to review pohcy and 
procedures. This was fohowed by on-site supervision from a fehow staff member during the 
new specialist's first week or two. Day-to-day, the program coordinator stayed in regular 
contact with career specialists to provide guidance on specific program issues (for example, 
whether a requested supportive service payment should be approved). Future Steps workers 
also received coaching on strategies for fulfihing such responsibilities as locating chents, 
accessing community resources, and advocating assertively for clients. Career speciahsts did 
not receive formal evaluations, but the placement center director and program coordinator 
did observe and discuss each worker's performance and provide periodic feedback. 

• Future Steps services centered on intensive, employment-focused case 

management intended to prepare participants for work and help them find and 

keep good jobs. 

The Future Steps design envisioned core services intended to help chents secure and 
maintain employment. These services included (1) an initial assessment of skihs and 
interests; (2) individuahzed job search, job assistance, and skihs enhancement plans; (3) help 
in overcoming personal and logistical barriers to employment through outside referrals, 
supportive service payments, and mentoring; (4) postemployment support; and (5) 
vocational training tailored to local employers' needs. (As discussed later, in the section on 
program implementation findings, the plan for using vocational training was not reahzed.) 

Chents referred to Future Steps by IDHS were assessed and quickly directed into 
activities intended to help them prepare for and find work. At initial meetings with a chent. 
Future Steps career specialists used standardized career assessment tools and checkhsts of 
skihs and barriers to gather details about the chent's employment history, job skills, and 
potential barriers to employment. After they got to know a chent better, career speciahsts 
engaged in case management activities, including job search assistance, making referrals to 
other service providers, disbursing supportive service payments, offering mentoring and 
counsehng, and providing postemployment support. These activities are described below. 

Job Search and Job Placement Assistance. Future Steps workers assisted chents in 
their job searches by helping them specify tasks that could lead to employment, monitoring 
their progress in completing these tasls, sharing information about available jobs, and 
contacting potential employers on chents' behalf. At each meeting, the Future Steps worker 
and the chent used an activity sheet to note tasks that should be completed, such as 
submitting an apphcation to a specific employer or writing a resume. In a tjq>ical week, the 
chent was expected to complete five tasks. To identify job opportunities, the career 
speciahsts used listings from tiie SCC placement center and local newspapers. Future Steps 
workers also carried out additional, community-specific job placement and job development 
efforts. T o varying degrees, workers relied on personal relationships with business ovmers 
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to get infomiation about available positions, cultivated new relationships with local 
businesses, or made cold calls to potential employers. 

Particularly capable Future Steps staff also advocated for chents by providing references 
to potential employers and aggressively marketing chents as good candidates for available 
positions. Career specialists would sometimes contact potential employers before a chent 
submitted an apphcation to verify the availability of the position and let the employer know 
to expect the chent. On occasion, a Future Steps worker would drive a chent to a potential 
employer for an apphcation or interview. 

Referrals for Services and T raining. Career speciahsts referred chents to local service 
providers for training, counseling, child care, and other services, as needed. The freguency 
and type of referral varied among staff members, as some cultivated close working 
relationships with particular service providers in their communities. 

Supportive Service Payments. Future Steps staff used supportive service payments to 
help meet clients' employment- related and personal expenses. Typical uses included 
payments for commuting, training, child care supphes, job-specific eguipment and clothing, 
and housing and utihty bills. The flexibility of supportive service payments allowed staff 
members to approve purchases of items as varied as a baby stroller, an electronic dictionaiy 
(for a chent with low hteracy skihs), and "stopgap" child care. One Future Steps worker's 
close connections with a local used car dealer helped her find dependable cars for chents, 
with loan arrangements that made the vehicles affordable. 

Supportive service payments were typicahy hmited to a total of $500 per chent, with 
exceptions made in some cases. For TANF chents, an additional $1,200 was available from 
ID HS, but these funds came with many restrictions (for example, the money could be used 
to pay for car repairs or fuel but not to purchase a vehicle). To maximize the resources 
chents could access. Future Steps workers sometimes supplemented IDHS payments with 
Future Steps funds. 

Mentoring. Future Steps staff offered chents mentoring and informal counsehng 
intended to help improve their work readiness and life skihs. Staff discussed a wide range of 
topics with individual chents as needed. These topics included transportation, family 
circumstances, appropriate workplace behavior, housing, and child care. Sometimes, guite 
practical issues were addressed. For example, one career specialist helped several of her 
chents budget for household expenses and taught them how to plan and shop for affordable 
home-cooked meals. Mentoring by Future Steps staff also offered an opportunity to 
encourage chents and improve their self-esteem, possibly enhancing their ability to find and 
keep employment. 

Postemployment Support. After a chent secured a job. Future Steps staff members 
were expected to contact tiie chent or employer at least eveiy two weeks for three months. 
The purpose of these contacts was to check on the chent's adjustment to the new position 
and, as needed, to offer help mediating any miscommunications or differences between the 
chent and employer. On-site visits to employers were expected within 30 days after a chent 
began work. In practice, the freguency of on-site visits varied among the Future Steps staff. 
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One staff member reported making two in-person job site visits per month for each of her 
employed clients, while others made site visits only for their more difficult chents. 

Although mediation between chents and employers did not occur freguently, it could be 
essential for helping chents retain their jobs when problems arose. For example, Future 
Steps workers described several cases when mediation was used: (1) to address an accusation 
that a client had stolen from a customer while on the job, (2) to resolve child care issues, and 
(3) to discuss a drop in a client's productivity. 

Future Steps chents could come back to the program if, for example, they became 
unemployed and needed help finding a new job. Clients' relatively long participation in the 
program— nearly 16 months, on average— reflected this extended support. Ongoing 
support for job advancement was also available to Future Steps chents, but staff members 
provided this kind of assistance to a minority of chents (as described in the section on 
clients' program experiences). 



The Future Steps Target Population 

The Future Steps target population consisted of three main groups: (1) people on 
TANF who were reguired to work as a condition of receiving assistance, (2) "able-bodied" 
food stamp recipients who were reguired to work as a condition of receiving assistance, and 
(3) low-income volunteers.^ The volunteer group could include people on TANF or food 
stamps who were not reguired to work, or those who received other types of support from 
ID HS, such as medical assistance. Volunteers also could include people not participating in 
any IDHS program but who hved in low- income households (under 200 percent of federal 
poverty guidelines). All chents had to be available and whhng to work at least 30 hours per 
week to be ehgible for Future Steps. 

Future Steps enrolled participants from all five counties in its service area. According to 
program records, most participants hved in Alexander or Pulaski Counties (33 and 27 
percent, respectively). These two counties had smaller populations than the other three, but 
they were also economicahy worse off (as described in the section on the program's context). 
Eighteen percent of participants hved in Union County, 14 percent in Massac, and 9 percent 
in Johnson. 

Information from basehne forms indicates that most members of the target population 
were female (76 percent) and that most were between ages 20 and 39 (77 percent) (Table 



1 When the demonstration program began, able-bodied food stamp recipients were not required to 
participate in Future Steps. Midway through the demonstration, however, able-bodied food stamp recipients 
became part of the mandatory Future Steps population. Although it is not possible to know precisely what 
fraction of the Future Steps population were volunteers, we can estimate, based on enrollment data, that 
volunteers made up between one-quarter and one- third of eligible Future Steps applicants. 
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II. 1).^ Most sample members also had children in their households (71 percent). Men 
without children (15 percent, not shown in table) made up a substantial minority of the 
Future Steps sample. Caucasians and African Americans made up about equal proportions 
of the sample group. We highhght other key characteristics of the sample below. 

• A small minority of sample members were receiving cash assistance at the time of 
referral to the program, although program designers had planned for TANF 
recipients to be a substantial share of the Future Steps target population. Receipt 
of food stamps was common. 

Current TANF recipients made up a small share of people referred to Future Steps- 
just over 15 percent— despite the program's original focus on this population. The low 
enrollment of people receiving cash assistance is consistent with the low number of TANF 
cases in the Future Steps service area (between 19 and 151 cases in each county at the outset 
of the demonstration). Nevertheless, it was anticipated that TANF recipients would make 
up a larger segment of Future Steps participants. On the other hand, most people referred 
to the program were on some t^e of pubhc assistance. Specifically, more than three- 
quarters of the sample population were receiving food stamps at the time of their referral (in 
addition to TANF for some sample members). Many of the food stamp recipients were 
considered "able-bodied" and were required to work to continue their ehgibihty for benefits. 

While current TANF recipients were a minority of the population referred to Future 
Steps, most people in the sample— 56 percent— had at least some history of receiving cash 
assistance. Nearly one-quarter (23 percent) of those referred were longer-term welfare 
recipients, on TANF or AFD C for more than two years. 

Sample members also generally reported low earnings from jobs in the year before their 
referral to Future Steps. More than 20 percent reported no earnings at all, and an additional 
45 percent reported that they had earnings under $5,000. Just over two percent of the 
sample population had earned more than $20,000 in the previous year. 



2 Although we focus our discussion in this section on baseline characteristics of the fuU Future Steps 
research sample, this table also highlights characteristics for program and control group members separately. 
The characteristics of the two groups were similar, and there were no systematic, significant differences 
between them. The significant differences present between the two groups are within the range of expected 
variation for a randomly selected sample. Moreover, as discussed in Chapter I, in the impact analyses, we 
controlled for baseline differences in the characteristics of the two groups using multivariate regression 
methods. 
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Table 11.1. Characteristics of the Future Steps Sample Population at Baseline (Percentages) 




Program 


Control 


All Sample 




Group 


Group 


Members 


Age 


Younger than 20 


7.0 


6.9 


7.0 


20 to 29 


47.6 


46.1 


46.8 


30 to 39 


30.4 


29.3 


29.8 


40 or Older 


15.0 


17.7 


16.3 


Average Age (Years) 


30.1 


30.3 


30.2 


Gender 


Female 


75.7 


77.6 


76.7 


Male 


24.3 


22.4 


23.3 


Race/Ethnicity 


Hispanic 


3.9 


3.0 


3.4 


White, Non-Hlspanic 


47.2 


44.1 


45.7 


Black, Non-Hlspanic 


47.6 


51.6 


49.6 


Other Race/EthnIcIty 


1.3 


1.3 


1.3 


Language 


Primary Language Is English 


99.4 


100.0 


99.7 


Transportation 


Has a Valid Driver’s License 


73.8 


70.7 


72.2 


Owns or Has Access to a Working Vehicle 


69.0 


65.1 


67.0 


Education 


No High School Diploma or GED 


29.5 


23.7 


26.6 


High School Diploma or GED 


44.0 


41.9 


42.9 


More than High School Diploma or GED 


26.5 


34.4** 


30.5 


Empioyment History 


Ever Worked for Pay 


98.4 


96.8 


97.6 


Currently Working for Pay 


19.9 


16.1 


18.0 


Worked During Past Two Years 


92.6 


90.2 


91.4 


Earnings in Prior Year 


None 


20.8 


20.6 


20.7 


$1 to Under $5,000 


44.8 


45.1 


44.9 


$5,000 to Under $10,000 


21.4 


21.0 


21.2 


$10,000 to Under $20,000 


10.7 


10.8 


10.8 


$20,000 or More 


2.3 


2.5 


2.4 
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Table 11.1 (continued) 





Program Group 


Control 

Group 


All Sample 
Members 


Never Received TANF or AFDC 


45.6 


42.3 


43.9 


Received TANF or AFDC 1 to 12 Months 


27.0 


22.4 


24.7 


Received TANF or AFDC 13 to 24 Months 


9.1 


8.0 


8.6 


Received TANF or AFDC 25 to 60 Months 


7.8 


12.8** 


10.3 


Received TANF or AFDC More than 60 Months 


10.4 


14.4 


12.4 


Current Public Assistance 


Currently Receiving TANF or AFDC 


15.6 


15.0 


15.3 


Currently Receiving Food Stamps 


79.9 


76.8 


78.3 


Currently Receiving General Assistance 


1.6 


1.9 


1.8 


Currently Receiving Housing Subsidy 


3.2 


6.4* 


4.8 


Currently Receiving SSI or SSDI 


5.5 


8.6 


7.1 


Household Characteristics 


Average Household Size (Number of People) 


3.1 


3.2 


3.1 


Child Under 18 In Household 


70.8 


71.5 


71.2 


Average Number of Children In Household 


1.4 


1.5 


1.5 


Average Age of Youngest Child (Years) 


4.9 


5.1 


5.0 


Youngest Child Less than 3 Years Old 


31.2 


27.1 


29.1 



Household Composition 


Single-Adult Household 


56.2 


56.3 


56.2 


Married or Partner Household 


16.6 


19.0 


17.8 


Other Multiple-Adult Household 


27.3 


24.8 


26.0 


Health 


Own or Other’s Condition Limits Activities 


18.3 


23.1 


20.7 


Sample Size 


313 


317 


630 



Source: Rural Welfare-to-Work baseline Information forms, compiled by Mathematica Policy 

Research, Inc. 



7*7*** Significantly different from zero at the .10/.05/.01 level, two-talled test. 
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• Some chamcteristics of the sample population, including prior work experience 
and education, suggest that many people referred to Future Steps were "work 
ready. " 

Much of the population referred to Future Steps appeared to have the experience and 
education needed to secure basic employment. Nearly all members of the sample group had 
an employment history. Ninety-eight percent of the sample population had worked for pay 
in the past, and more than 90 percent had done so within the past two years. Not 
surprisingly, only a small fraction (about one-fifth) were working for pay at the time of their 
referral to Future Steps. About three-quarters (73 percent) of the sample had at least a high 
school diploma or G E D , and nearly a third (31 percent) had education beyond high school. 

• Many sample members also faced potential obstacles to work, including 
responsibility for young children or a health problem or disability. 

Single parents made up a large portion of the sample population, so many people 
referred to Future Steps probably needed access to reliable child care to secure a job and stay 
employed. More than hdf of the sample members were single parents, and nearly 30 percent 
had a child younger than age 3. The average age of sample members' children was 5. 

A substantial minority of the sample population— more than 20 percent— reported that 
they or another member of their household had a health problem that hmited their ability to 
work, attend training, or go to school. These problems included a medical issue or physical 
disabihty, emotional or mental health conditions, and substance abuse. 

• At least one-third of the sample faced barriers related to transportation. 

Lack of reliable transportation was a potential impediment to employment for a sizable 
minority of the Future Steps population. As described in the next section, the pubhc 
transportation infrastructure was extremely hmited in the Future Steps service area. Program 
participants who had to commute to work would typically need access to a car or a rehable 
ride with friends or relatives. On baseline forms, however, almost 30 percent of the sample 
reported that they did not have a valid driver's hcense, and 33 percent did not own a car or 
have daily use of a car. 



Southern Illinois: The Economic, Policy, and Service Context 

The rural context of Future Steps shaped the economic opportunities and resources 
available to sample members. While economic conditions in the Future Steps service area 
were generahy weak, local residents could draw on personal networks and a limited number 
of social service providers for assistance when necessary. In this section, we describe the 
economic, policy, and service environment in which Future Steps operated. 
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• AU Gve counties in the Future Steps sendee area faced significant economic 
challenges, including high poverty and a limited economic base. 

Economic data highlight the generally weak conditions in the area served by Future 
Steps (Table II.2). County-level poverty rates ranged from 13.3 to 23.6 percent in 2001 (the 
beginning of the evaluation period). These rates all exceeded the statewide figure of 
10.1 percent. Median household income in the target counties was $25,231 to $33,489, weU 
below the statewide median of $46,991. The counties' unemployment rates presented a 
more mixed picture. Unemployment in Johnson, Massac, and Union counties was near the 
statewide average of 5.4 percent, but rates in Alexander and Pulaski counties (8.3 and 
7.9 percent, respectively) were much higher. 



Table 11.2. Key Economic Indicators for Counties in the Future Steps Service Area 



(2001 Figures) 



Area 


Poverty Rate 
(Percentages) 


Unemployment Rate 
(Percentages) 


Median Household 
Income (Dollars) 


Alexander County 


23.6 


8.3 


25,231 


Johnson County 


14.4 


5.8 


33,489 


Massac County 


13.3 


5.4 


32,462 


Pulaski County 


20.9 


7.9 


25,317 


Union County 


13.4 


6.0 


31,805 


State of Illinois 


10.1 


5.4 


46,991 



Sources: U.S. Census Bureau, U.S. Department of Labor. 



The local economy slowed in the second half of the demonstration. Statewide 
unemployment had risen to 6.7 percent by 2003, and unemployment rates in four of the five 
counties served by Future Steps showed similar increases. (The exception was Massac 
County, whose economy remained relatively strong.) Future Steps and ID HS staff reported 
that fewer entiy-level jobs were available in the local area and that some cheats had to travel 
further to take advantage of job opportunities. 

Economic resources differed to some extent within the Future Steps service area. 
Massac and Union counties were moderately prosperous, compared to the other three. A 
riverboat casino in Massac County, for example, provided tax revenue and promoted 
additional businesses, such as restaurants and hotels. Manufacturing jobs at chemical and 
cement factories were also available in the area. Union County benefited from tourism and 
was home to companies that produced baked goods and shoes. In contrast, Alexander, 
Johnson, and Pulaski counties had few large employers, and even small businesses were 
scarce in some cities in these counties. Johnson County, with the smallest population of the 
five, had an especially hmited economic base, centering on three state prisons. 
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Entry-level employment opportunities in the area commonly included jobs at local 
nursing homes, housekeeping and other service work, and cashier or warehouse positions at 
large retailers. A temporary agency serving companies in nearby Cape Girardeau, Missouri, 
also employed some residents of the area. According to Future Steps and ID HS staff, many 
available jobs for entry-level workers were part-time (less than 28 hours per week). 



• The Illinois TANF program emphasizes chents' mpid attachment to work. 

Ilhnois TANF policies during the evaluation period reflected a "work first" philosophy. 
People not covered under the program's few exemptions were reguired to engage in work 
activities for 30 hours per week (35 hours for two-parent households), and a 30-day job 
search and waiting period was enforced before TANF applications could be approved. At 
the discretion of local office administrators, a small number of chents could participate in 
education and training activities to meet part of their work reguirement. 

The state's TANF program also provided positive incentives for paid employment. A 
generous earned income disregard and transitional eligibihty for medical coverage and child 
care helped make employment financially advantageous for clients. In addition, months in 
which clients fuMhed their work reguirements through paid employment did not count 
toward the 60-month lifetime hmit for cash assistance. (Clients were shifted to state- funded 
assistance during these months.) 

• The TANF caseload declined substantially in the Future Steps sendee area after 
welfare reform hut began to rise again during the demonstration period. 
Throughout the period, individual WHS workers maintained large caseloads. 

As in many other states, the TANF caseload in Ilhnois dechned sharply after welfare 
reform. Caseloads in the counties Future Steps served initially fohowed this downward 
trend. ID HS administrators reported a rise in the number of TANF cases in the second half 
of the demonstration period, however. This increase in the TANF caseload coincided with 
an economic downturn. The fraction of Future Steps sample members on TANF at the end 
of the 18-month fohow-up period did not increase, however. 

Individual counties in the Future Steps service area had a relatively smah number of 
TANF cases at the beginning of the demonstration, but caseloads of IDHS workers often 
were large. Many ID HS workers handled caseloads that included recipients of other types of 
assistance, such as food stamps, and individual staff members were responsible for as many 
as 400 chents. This made it difficult for ID HS workers to provide individuahzed services, as 
they spent much of their time determining or recertifying eligibihty, updating chent records, 
and responding to chent inguiries. In the second year of the demonstration, caseload sizes 
for IDHS line staff increased by 30 to 40 percent as overall TANF participation increased 
and after agency budget cuts resulted in a reduction of the number of caseworkers in the 
five-county area. 
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While IDHS workers were limited in their ability to provide individualized services, 
members of the evaluation's control group did have access to various forms of employment 
assistance through the agency. IDHS made hstings of job openings available to clients and 
offered job readiness and job search workshops (including resume- writing assistance). 

• Almost all sample members noted that they had community connections and 
could get help horn Mends, family members, and neighbors if necessary. Most 
indicated that community-based social service agencies existed in their 
home areas. 

Survey data indicate that people in the evaluation's sample group felt familiar with, and 
connected to, their local area.^ For example, control group members tended to be longtime 
residents (about 14 years, on average), and a large majority (73 percent) felt "veiy connected" 
or "fairly connected" to their communities (not shown). Members of the evaluation sample 
also felt they could access different types of support if needed. Almost all control group 
members reported that they had family, friends, or neighbors they could turn to for help 
with such needs as transportation, shelter, meals, or child care, and more than three-quarters 
said that they had received some type of help from family or friends during the 18-month 
follow-up period (Table II. 3). More than 90 percent said that a community organization 
providing such services as a food pantiy, crisis hothne, thrift store, or family assistance was 
available in their area, and more than 60 percent had accessed these services at some point 
during the foUow-up period. 

Even with support available from family, friends, and community-based organizations, 
life in the five Future Steps counties posed challenges for some people. In focus groups, 
members of the evaluation's sample group shared observations regarding such issues as 
limited transportation and the tight social networks that can shape employment 
opportunities (see text box). 

• In addition to IDHS, nonprofit and government organizations provided social 
services and employment assistance in the five-county region, but significant 
service gaps remained, particularly in the area of transportation. 

Education, health, and workforce development organizations offered assistance to 
people living in the Future Steps service area. Such services were available to all members of 
the evaluation's sample group, both program and control. SCC was a primaiy provider of 
education and worlrforce development services in the region. In addition to associate's 
degree programs, the college made available such services as professional development 



3 Survey data were collected 18 months after random assignment. Hence, the outcomes collected through 
the survey could have been influenced by the program. However, we present these survey data here only to 
illustrate the program's context. G iven the timing of the survey data collection, we focus on the control group 
data, although no systematic differences were found between program and control group members in their 
community connections and access to, and use of, support networks. 
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Table 11.3. Access to, and Use of, Support Networks Among Sample Members 

Program Control 

Characteristic Group Group 



Has Access to Help from Family, Friends, or Neighbors (if Needed) 

With transportation 

With lodging 

With food 

With money 

With child care 

With any of the above 

Received Help from Family, Friends, or Neighbors in the Past 18 Months 

Availability of Community Organizations in the Sample Member’s 
Home Area 

Food pantry or soup kitchen 
Crisis hotline or walk-in center 
Thrift, Goodwill, or Salvation Army Store 
Church or family service organization 
Any of the above 

Used Services of a Community Organization in the Past 18 Months 

Sample Size 



90 


91 


91 


89 


92 


92 


78 


82 


83 


84 


97 


96 


76 


77 


76 


80 


52 


59 


64 


72 


78 


80 


93 


93 


63 


62 



252 



268 



Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy 

Research, Inc. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-tailed test. 



seminars and training for small businesses. Training services were also available through the 
Regional Office of Education, a state agency, which offered certificate courses for nursing 
assistants, GED classes, and hteracy tutoring. The Shawnee Development Council, a 
nonprofit community action agency and Workforce Investment Act contractor, operated a 
One-Stop Center in Cairo, offering services for adult, youth, and dislocated jobseekers. 

Physical and mental health care services were available through community-based 
organizations, hospitals, and government agencies. The Southern Seven Health 
Department/ Head Start, for example, was a key provider of pubhc health services to low- 
income residents of the five-county region. Several community and regional hospitals 
provided inpatient care, and mental health and substance abuse counsehng were available 
through small nonprofits. People who were disabled could receive help securing 
employment through the IDHS Office of Rehabilitation Services, which provided diagnostic 
assessments, physical rehabilitation, workplace accommodations, provision of eguipment, 
and on-the-job coaching. 

Despite the efforts of the organizations and agencies mentioned above, significant 
service gaps remained in the region. Eor example, a substantial proportion of the sample 
population had circumstances that suggested a need for pubhc transportation. On the 
18 -month follow-up survey, less than half of control group members reported that they were 
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Life in a Rural Area: Transportation Challenges 
AND Tight-Knit Communities 

In focus groups, members of the evaluation's sample group offered insights on 
the disadvantages and advantages of hving in a rural area. Conditions such as hmited 
transportation and tight-knit social networks had practical effects on the day-to-day 
lives of these residents of southern Ilhnois. 

Some focus group participants described just how difficult life could be without 
adequate transportation. "There [are] no jobs here in Illinois," one participant 
commented. "Everybody here in Illinois has to travel so far, as far as Kentucky, to 
find a job. And not everyone has the transportation. And that's hard." According to 
another participant, "If you don't have no transportation and don't have no baby- 
sitter, you don't get no job. Because those are the main things you got to have." 
Some people were kept from school. "I wanted to go to Shawnee Community 
College but I don't have a car yet," one participant said. "I got to get my hcense. I 
ain't got no transportation from here to Shawnee." Others were kept from work: 
"It's not like I don't want to work. I really do want to work, but there's nowhere 
close around to go to work. If you don't have any transportation in southern Illinois, 
you can't even get to the grocery store in your town." 

For some residents of tight-knit rural communities, social networks were 
important in their efforts to find employment. Finding and keeping a job sometimes 
required the support of family and friends helping with chUd care, transportation, and 
job searches. "I got lucky because my little brothers and sisters helped me watch my 
baby," one participant commented. "My momma helped me. But some people 
don't have that help." When asked how they obtained their jobs, several focus group 
members attributed it to family or people they knew in the community. "My 
daughter got the job for me," a participant explained. "She put my name in and they 
called me the next day and [I] went to work that night." 

Those without this community support saw the downside to tight-knit 
communities. New residents felt isolated in their job search without personal 
connections to aid them. One such newcomer felt that "if you're from out of the 
area and you move into an area like this, you're an outcast. They don't want to hire 
you because it's not the way; it's too old-fashioned. To me, smaller communities ... 
hire from within." Another concurred, saying, "Everything around here is family- 
oriented. They keep the money amongst themselves. If you know the right person, 
you can get hired in." 
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always able to travel to a needed place (Table 11.4).^ In addition, when asked about 
commuting to an area with better jobs, control group members reported, on average, that 
such a commute would take about 45 minutes (one way), and over two-fifths indicated that 
such a hypothetical commute would be difficult, typically because of a lack of transportation 
or driver's hcense. At the time of the follow-up survey, working control group member 
commuted 26 minutes, on average, to their jobs (Appendix C, Table C.5). A local economic 
development group hoped to create a regional pubhc transportation infrastructure in 
southern Ilhnois. This effort was at an early stage, however, and pubhc transportation 
options were extremely hmited or nonexistent in most parts of the five-county region. 
Other service shortages related to adeguate low- income housing and dependable, accessible 
child care. 



Table 11.4. Transportation Access and Issues 



Characteristic 


Program 

Group 


Control 

Group 


How Frequently Sample Member Is Able to Travel to Needed Places 


Almost always 


50 


46 


Often 


21 


25 


Sometimes 


20 


24 


Hardly ever or never 


9 


6 


Average Travel Time from Home to Key Places (One-Way, by Car) 
(Minutes) 


To Public Assistance Office 


12.6 


12.4 


To supermarket 


19.2 


19.4 


Hypothetical Commute to Area with Better Jobs (Average Estimated 
Time of Hypothetical One-Way Commute to Area with Better Jobs, 


in Minutes) 


48 


45 


Sample Member Believes That (S)he Would: 


Need to travel to another area to get a good job or better job 


89 


91 


Face difficulty commuting to another area with better jobs 


47 


43 


Sample Size 


252 


268 



Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy 
Research, Inc. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-talled test. 



4 For the same reasons stated in footnote 3, we present control group data here. No significant 
differences were found between the program and control groups in these transportation data. 
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Key Findings on Future Steps Implementation 

Implementing a welfare-to-work program in a rural context poses challenges related to 
covering a large, dispersed service area, recruiting and supporting gualified staff, and 
maintaining a commitment to a new program in an environment of limited resources and 
competing priorities. In this section, we highlight both successes and challenges in the 
implementation of Future Steps. 

• A strong collaboration between WHS and SCC helped smooth the initial stages 
of Future Steps, but later organizational budget pressures weakened program 
operation. 

Administrators at SCC and IDHS perceived the partnership between the two 
organizations to be strong. The partners' history of collaboration on Advancing 
Opportunities provided a solid foundation for implementing a new program. Moreover, 
some former Advancing Opportunities staff members, including the program director, took 
on similar roles in Future Steps. Existing personal relationships between administrators at 
SCC and IDHS and co-location of staff also helped facilitate communication about 
expectations for referrals and service delivery to enrolled chents. 

Channehng Future Steps services through SCC produced several advantages for the 
program. The positive reputation of the community college helped reduce the stigma chents 
might experience as participants in a welfare-to-work program. According to program staff, 
employers also were more hkely to respond to inguiiies from Future Steps when they were 
told of the program's affiliation with the community cohege. The resources of the 
community college's placement center benefited the program as weh. Future Steps career 
speciahsts had easy access to the extensive job hstings that the placement center maintained, 
along with such additional career resources as assessment tools, resume-writing assistance, 
and career- planning guides. 

Both IDHS and SCC faced financial pressures in the second year of the demonstration. 
IDHS responded to budget cuts by reducing frontline staff (and increasing individual 
caseloads) in local offices. SCC administrators were similarly concerned about the cohege's 
budget. This issue may have diverted administrators' focus on Future Steps and indirectly 
affected program expenditures— for example, not ah available funds for supportive services 
were used during the program. The different organizational missions of ID HS and SCC may 
also have affected the implementation of Future Steps in its second year. Although IDHS 
and SCC had a history of collaboration, TANF and food stamp recipients were not the 
cohege's core constituency. In a context of limited resources, SCC may not have prioritized 
operating Future Steps at a consistently high level. In the second year of the program, for 
example, the director's time commitment to Future Steps feh, and the number of career 
speciahsts shrank from five to three. 
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• The core program model was implemented largely as intended, and Future Steps 
appeared to offer substantially more individualized support to clients than was 
available otherwise. 

In general, Future Steps staff successfully provided the core services of the program 
model— assessment, job search assistance and followup, and help overcoming barriers to 
employment. (Data from the Future Steps Information System [FSIS], described in the 
section on client experiences, offer details on the type and freguency of services dehvered.) 
After an expansion in the target population to include food stamp recipients and low-income 
volunteers, the program also enrolled as many clients as planned— a total of 313 during its 
two years of operation, exceeding the enrollment goal of 300. 

Future Steps appeared to offer more tailored and intensive services for low-income 
people than other providers in the area. In part, this was a result of the small caseloads 
carried by Future Steps career speciahsts— approximately 15 to 35 active chents, compared 
to the caseloads of 200 to 400 for IDHS workers. (Future Steps caseloads also included 
"transitional" chents who had secured and retained employment and no longer needed 
intensive services. These clients represented additional cases totahng up to 50 percent of the 
active caseload.) Future Steps workers also individuahzed services through judicious use of 
supportive service dollars, mentoring, and referral to suitable job opportunities. In focus 
groups, chents offered examples of the assistance they received to address individual 
obstacles, including money for clothing, gas, car repairs and insurance, and educational 
testing fees. 

• Program leaders worked creatively to oversee staff across a dispersed service area, 
but they found it challenging to manage staff with varied skill levels. 

To provide services over a large area. Future Steps career speciahsts worked in separate 
locations from the program director and coordinator. Program leaders worked creatively to 
monitor and support staff from a distance, communicating freguently by email and 
telephone and reviewing case notes entered into the FSIS. 

Future Steps career speciahsts varied in their skih levels, however, and program 
administrators sometimes were challenged to ensure that high-guality services were provided 
consistently. For example, some staff members were more assertive than others in 
advocating for chents, developing employment opportunities, and leveraging available 
community resources. Career specialists also differed in their famihaiity with the 
communities they served. Those who had lived in the local area for some time and 
maintained wide personal networks often were able to use these connections to their chents' 
benefit. Since staff worked without on-site supervision and received only brief training, it 
was difficult for many to improve their performance. 
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• Staff turnover during the second program year led to short-teim breaks or 

slowdowns in sendee dehveiy. 

Future Steps experienced substantial staff turnover in its second year, with some 
consequences for program operation. The program coordinator left early in the second year 
of Future Steps, and the program director departed several months later. A senior-level 
college administrator assumed responsibihty for directing Future Steps but did not have as 
much time available to devote to the program as the original director. Although an 
experienced career speciahst replaced the coordinator, the loss of the program's key original 
sMf members appeared to hamper program development and result in less oversight of the 
career specialists during the second half of the demonstration. 

Several career speciahsts also left the program in the second year. By the end of the 
two-year evaluation period, the number of speciahsts had dropped to three, with some 
speciahsts serving more than one county. Caseloads also dechned toward the end of the 
evaluation period, but service dehveiy appeared to become less intensive as some staff 
members began covering larger geographic areas. 

In general, recruiting and retaining appropriately qualified staff members appeared to be 
a challenge for Future Steps administrators. Program leaders commented that it was difficult 
to find people with the combination of professional skhls and community famiharity needed 
to perform weh in the position. In addition, most career specialists were hired as part-time 
employees and did not receive benefits, making the position less attractive. 

• One element of the program model— employer-focused job readiness and 

vocational training— did not materialize. 

Although Future Steps plans included employer-focused training opportunities for 
program chents, this part of the program design was not reahzed during the demonstration 
period. Program designers had intended to enhance clients' work readiness and job-specific 
vocational skills by working with local employers to develop customized vocational trainings, 
which would prepare chents for, and connect them to, good jobs in the local area (that is, 
jobs that offered hving wages and a chance for benefits and advancement). 0 ne potential 
business partner was a fruit packaging company; the Future Steps program director hoped to 
train participants for work on the factory floor. Program administrators also considered 
offering training for retail positions. 

Several factors appeared to contribute to the program's inability to offer focused 
training in partnership with employers. This element of the program model was not in place 
when Future Steps launched. This ambitious element would have required substantial staff 
time to implement. G iven that the program director, who had held primaiy responsibihty 
for program development, was a part-time Future Steps employee who departed midway 
through the demonstration, following through on initial plans was difficult. Meanwhile, 
because of increased fiscal pressures within SCC, program administrators focused on other 
issues. In addition, a weak economy and difficulty partnering with the local workforce 
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investment agency made it challenging to begin recruiting employers to participate in 
the effort. 

Client Experiences in Future Steps 

Future Steps successfully delivered a substantial amount of assistance to most of the 
people who enrolled in the program. However, cheats' experiences do appear to vaiy 
somewhat based on when they enrobed in the program and whether they became employed. 
An analysis of data from the FSIS and from participants' feedback during focus groups 
offers insights on the intensity and duration of clients' participation, the types of services and 
level of assistance they received, and the extent to which they entered employment 
and training. 



• Most clients met or intemcted with Future Steps career specialists many times 
over an extended period. While many received a high level of sendee, a substantial 
minority received few sendees and had little contact with their caseworker. 

On average, clients had 25 contacts with a Future Steps staff member— many of which 
comprised multiple activities, such as employment assistance, discussions or mentoring on 
personal and family matters, and assistance with service needs or referrals (Table II. 5). 
(Only those contacts where services were actuahy delivered are included in this total.) 
Contacts occurred in person and by telephone, with the former being more common. 
Almost ah clients who were referred to Future Steps (93 percent) had at least one contact 
with a career speciahst, and nearly half (49 percent) had more than 20 contacts. Although 
individual contacts generally were brief— about 11 minutes— total contact time averaged 
about 11 hours per client. The average length of enrollment among Future Steps clients was 
nearly 16 months, and 70 percent of clients were in the program for more than a year, 
although most services were delivered during the first several months after a client's 
enrohment.'’ 

Future Steps clients had diverse experiences in the program, with some receiving a 
considerably more intensive level of service than others. Nearly two-fifths (38 percent) 
received services at a high level, meeting at least two of the following three criteria: (1) more 
than 25 service contacts with their career specialist, (2) more than 12 hours of contact time, 
and (3) more than $400 in supportive service payments (not shown). This level of 
participation suggests that these chents likely received a full range of services over an 
extended period, as the program intended. 



^ We estimated clients' length of enrollment in the program as the duration of time between the date of 
their enrollment into the program and the date on which they received their last program service. In almost aU 
cases, clients' date of enrollment was the same as, or within a few days of, their date of random assignment. 
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Percentage 



Service Contacts (All Service Types) 

At least one contact 93 

More than 10 contacts 72 

More than 20 contacts 49 

Average number of contacts 25 

Service Contacts (Specific Service Types) 

At least one mentoring/counseling discussion 91 

More than 10 mentoring/counseling discussions 60 

At least one employment help contact 83 

More than five employment help contacts 40 

At least one referral 42 

Three or more referrals 17 

At least one mediation with employer or other agency 9 

Average Total Contact Time (in Hours, Among Clients with at Least One Contact) 10.6 



Sample Size 313 



Source: Future Steps Information System, compiled by Mathematica Policy Research, Inc., as part of 
the Rural Welfare-to-Work Evaluation. 

In contrast a smaller, but still substantial, proportion of clients participated little in the 
program or not at aU. Nearly one-fifth (18 percent) met at least two of the following three 
criteria suggesting a low level of involvement: (1) fewer than five service contacts with their 
career specialist, (2) less than three hours of contact time, and (3) $100 or less in supportive 
service payments. Some of these chents may have been unwUhng to meet with Future Steps 
staff because they did not desire services or feel they needed them, or they may have been 
difficult to contact after referral to the program. In addition, career specialists with limited 
training or experience may not have been successful in keeping less motivated chents 
engaged in the program or in targeting Future Steps services to help maintain 
clients' participation. 

• Chents who entered Future Steps during the Gist half of the evaluaUon period 
appeared to receive more intensive services, as weU as a broader set of services. 

Clients who entered Future Steps during the first half of the demonstration had more 
contacts with staff and more contact time than those who enrobed during the second half.® 



® The program was implemented during the two-year period from July 2001 to September 2003, and 
random assignment and program enrollment were conducted during the 18-month period from July 2001 to 
D ecember 2002. Clients who enrolled during the first and second halves of the demonstration were those who 
were randomly assigned during the first and second 9-month periods, respectively, of the 18-month enrollment 
period. 
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Comparing clients' first six months of participation in Future Steps, those entering the 
program during the first half of the enrollment period averaged about four more contacts 
with staff than those entering during the second half of the enrollment period (Table II. 6). 
In addition, the percentage of clients in the earlier group who had more than 10 contacts 
with program sMf in their first six months (78 percent) was larger than for the later group 
(66 percent). The total amount of contact time also was higher for the earher group, by 
almost three hours, on average. These differences in service receipt and intensity are 
consistent with the decrease in implementation guality during tiie second half of 
the demonstration. 

It also appears that certain types of services were offered less freguently to chents 
enrolling in the second half of the demonstration period. For example, chents in the later 
group were much less likely to receive a referral to another service provider (Table II.6). In 
addition, although Future Steps staff members performed employer mediation on behalf of a 
minority of chents enrobed in the first half of the demonstration (16 percent), none of the 
chents enrobing later received this service (not shown). These patterns suggest that career 
speciahsts were less fully engaged with chents in the later part of the demonstration. 



Table 11.6. Service Use and Participation During First Six Months After Enrollment, by 



Enrollment Period 





Enrolled in 
First Half of 
Demonstration 


Enrolled in 
Second Half of 
Demonstration 


Service Contacts (All Service Types) 


At least one contact (percentage) 


95 


88 


More than 10 contacts (percentage) 


78 


66 


More than 20 contacts (percentage) 


50 


39 


Average number of contacts 


22 


18 


Service Contacts (Specific Service Types) (Percentage) 


At least one mentoring/counseling discussion 


95 


86 


Ten or more mentoring/counseling discussions 


64 


49 


At least one employment help contact 


82 


79 


Five or more employment help contacts 


39 


21 


At least one referral 


45 


25 


Three or more referrals 


20 


6 


At least one supportive service payment 


76 


68 


Average Total Contact Time (in Hours, Among Clients with 


at Least One Contact) 


8.1 


5.2 


Sample Size 


174 


139 



Source: Future Steps Information System, compiled by Mathematica Policy Research, Inc., as 
part of the Rural Welfare-to-Work Evaluation. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-tailed test. 
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Figure 11.1. Topics of Counseiing/Discussion Contacts 

Percentage of Clients Having at Least One Discussion 




Discussion Topic 

Source: Future Steps Information System, compiled by Mathematica Policy Research, Inc., as part of the Rural Welfare-to-Work Evaluation. 



• Future Steps employment services emphasized general job search advice. Job 
placement and job advancement assistance was less common. 

Most Future Steps clients received some type of employment assistance, but staff 
members were more likely to provide job search advice than help with placement or 
advancement. Career speciahsts helped more than 80 percent of their chents with their job 
search, primarily by identifying job opportunities. A much smaller proportion of clients 
(38 percent) received job placement assistance, such as having a career speciahst contact an 
employer to help arrange an interview. Finally, 12 percent of chents received advice from 
Future Steps staff on how to advance in employment after they had secured jobs. 

• Future Steps staff addressed logistical and personal issues in their discussions 
with chents. A minority of chents were referred to outside providers for other 
services. 

Career speciahsts helped many Future Steps chents address personal and logistical issues 
through mentoring or informal counseling discussions. More than 9 in 10 chents 
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(92 percent) participated in at least one mentoring discussion with their career speciahst. 
While involved in the program, the average client had 23 mentoring discussions with career 
speciahsts about one or more personal, family, or logistical issues in his or her hfe. 

Transportation and family issues were the most common topics addressed in 
discussions between career speciahsts and clients (Figure II. 1). More than four-fifths of the 
client population discussed transportation at least once, and many raised the issue guite a few 
times. (Of those clients who discussed transportation with a career specialist, nearly 60 
percent did so more than 10 times.) This freguency is not surprising, given the rural context 
of Future Steps. More than a third of Future Steps clients (37 percent) raised family issues, 
such as conflict with a spouse or partner, in conversations with career speciahsts, and similar 
proportions had discussions about child care and housing. Just a guarter of ah Future Steps 
clients spoke with career speciahsts about workplace behavior and ethics, reflecting a limited 
focus in these discussions on job readiness issues. 

Although Future Steps staff members could make referrals to other providers to help 
address employment barriers, they did not do so for most chents. Career specialists referred 
41 percent of chents to other service providers for assistance, usually training. About one- 
thW of all chents (32 percent) received a referral for training or education. Much smaher 
proportions received referrals for needs such as mental health care (six percent) or substance 
abuse treatment (two percent). 

• Clients received an average of $297 in supportive service payments, often toward 
their transportation expenses. Staff assisted many clients beyond the $500 soft 
cap, but almost Jin 10 clients received no supportive service payments at all. 



Flexible supportive service payments were an important part of the Future Steps 
program model, and most chents used this assistance. Almost three-guarters of all Future 
Steps chents (72 percent) received at least one supportive service payment, and 29 percent of 
chents received five or more (Table II.7). Chents received a total of $297 in payments, 
on average. 

Future Steps staff exceeded the “soft cap" of $500 in supportive service payments for 
more than a guarter of their chents (26 percent), indicating that career specialists sometimes 
maximized the flexibihty of the payments to meet the needs of individual chents. However, a 
similar proportion of chents (28 percent) received no payments at all. 

Although supportive service payments were designed to be flexible and cover a wide 
variety of needs, funds were most often used simply to help reimburse chents for 
transportation costs. Among those chents receiving supportive service payments, 67 percent 
received at least one payment for transportation (Figure II. 2), and nearly 60 percent of 
chents using supportive service dollars received more than three transportation-related 
payments. Expenditures on eguipment or clothing were also common; 34 percent of chents 
received a payment to help purchase such items. Much smaller proportions of chents used 
supportive service payments for housing (seven percent), utilities (five percent), or education 
(three percent). 
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Program Group 



Receipt of Payments 

Received at ieast one payment (percentage) 72 

Received five or more payments (percentage) 29 

Average number of payments received 3.5 

Vaiue of Payments 

Average totai amount among aii ciients (doiiars) 297 

Average totai amount among ciients who received payments (doiiars) 41 1 

Average amount per payment (doiiars) 99 

Received payments totaiing $250 or iess (percentage) 56 

Received payments totaiing $251 to $500 (percentage) 19 

Received payments totaiing more than $500 (percentage) 26 



Sample Size 313 



Source: Future Steps Information System, compiied by Mathematica Poiicy Research, Inc., 
as part of the Rural Welfare-to-Work Evaluation. 



• Service use was more substantial for clients who became employed than for those 
who did not Services were most intensive for employed clients before they 
secured a job and during the fust three months after they secured a Job. 

According to FSIS data, 65 percent of Future Steps clients became employed at least 
once while they were enrolled in the program. Among clients who became employed, nearly 
two-fifths (39 percent) were employed at least 90 days. Job turnover was common among 
Future Steps clients, however. FSIS data indicate that more than a fifth (22 percent) of 
employed clients changed jobs once, and 32 percent changed jobs two times or more while 
enrolled in the program. 

Among Future Steps clients who did not become employed, a large majority were 
engaged in the program, but they tended to receive less freguent or intensive services than 
clients who did secure employment. More than four-fifths (81 percent) of chents who 
remained unemployed had at least one contact with a career specialist. However, they 
averaged only 9 total contacts, compared with 44 for clients who found jobs. Similarly, 
among clients who remained unemployed, fewer received supportive service payments, 
compared with clients who became employed (48 versus 95 percent). 

The intensity of chents' participation in Future Steps was most active before they 
secured employment and during the first three months after becoming employed. Chents 
who became employed had an average of four service contacts per month with their career 
speciahst before getting a job, over a period averaging seven months. In the first three 
months after becoming employed, chents averaged three service contacts per month with 
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Figure 11.2. Uses of Supportive Service Payments 

Percentage of Clients Receiving at Least One Payment 




Transportation Equipment or Housing Utilities Education or Child Care 

Clothing Training 

Purpose of Payment 



Source: Future Steps Information System, compiled by Mathematica Policy Research, Inc., as 
part of the Rural Welfare-to-Work Evaluation. 



their career specialist. After that, they averaged just under 1 contact per month. These 
findings suggest that the program played an active role in providing postemployment 
support during the first three months after chents secured employment. After that, services 
were generally delivered on an as- needed basis, usually at the client's reguest. 

• Clients reported that the program strengthened their motivation and self- 
confidence and gave them a sense of control over their lives. However, some 
expressed concerns about the program's ability to provide effective assistance in a 
weak job market 

In focus groups, many Future Steps clients expressed the opinion that program services 
were valuable and had helped them better their situations. Chents described the importance 
of tangible benefits of the program— for example, supportive service payments to purchase 
work clothes or pay for car repairs— as weh as the difference the program made in 
improving their motivation and confidence to gain employment and overcome obstacles. 

Many chents noted that the presence, encouragement, and ongoing emotional support 
of the Future Steps career speciahsts were important factors in their success. As one client 
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expressed, “It's just somebody being there, knowing [and] not thinking that they [are] better. 
Just being there, period." Others explained, "When you have someone backing you hke [the 
career specialist], you feel like you can do anything" and "Y ou don't want to let her down so 
you push yourself to do it." These chents felt that their career specialist was interested in 
them and had helped give them the confidence to set goals for themselves, overcome 
hurdles, and move toward economic independence. 

Nevertheless, some clients also conveyed the view that the program was limited in its 
abihty to help them address their considerable long-term obstacles. One participant, for 
example, felt that the usefulness of supportive service payments was constrained by the 
general hmit of $500. "Five hundred dollars for everything isn't even insurance for two or 
three months for a vehicle," she explained, "much less the cost of child care." 

Many clients drew attention to the lack of readily available employment opportunities in 
the area, and some were disappointed that Future Steps provided limited services to address 
this. For example, one participant in the focus groups explained that the better jobs are "$10 
[or] $15 an hour jobs, but it's an hour and a half to get there." Most employed chents in the 
focus groups attested that the program services did not directly result in their employment; 
rather, they had found their job through friends, neighbors, or family. One client noted that 
Future Steps "needed to have more jobs... . They should have [had] the contacts... . We're 
coming to [Future Steps] to get a job and [the program is] still trying to send us 
somewhere [else]." 

Focus group participants expressed mixed opinions about the value of the job retention 
services Future keps provided. Some who had secured a job felt that Future Steps could do 
httle to help them keep it or to get a raise or promotion: "Once you're hired, the program 
has nothing to do with [helping you keep a job]." Still, these unenthusiastic sentiments were 
not uniform and varied across clients, often by the county in which they lived. Some other 
clients were positive about the employment services Future Steps provided. One said, "If 
[the career specialists] see [that] you're interested, then they redly do the extra." Another 
noted that the best part of the program was the assistance her worker provided through 
"one-on-one calling [of prospective employers] and helping [her] to get a [specific] job." 
StiU another chent shared that her career speciahst had visited her at work "two or three 
times, just to check on me [and] make sure if I needed anything." 



Costs OF Future Steps 

An assessment of program costs is important for understanding program 
implementation, making recommendations about program improvements, and guiding 
future planning and development efforts. As part of the evaluation, we developed an 
estimate of the total annual operating costs of the Future Steps program, as well as a unit 
cost estimate for the average participant the program served. Here, we highlight key findings 
from our program cost study. Appendix B provides an analysis of costs by program 
component. 
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• The cost to operate Future Steps during a one-year period totaled $333,214. 

We constructed an estimate of the aggregate cost of operating the program during a 
one-year, steady-state period. The cost analysis period represents a typical, ongoing year of 
program operations— one with relatively stable services, staffing, and client flows.’ We 
measured the market value of all resources used to operate the program and dehver services 
to participants during the cost period, including “off-budget" expenses that were donated, 
covered by other programs' resources, or absorbed by an organization's general 
administrative structure.® Such a market-based approach provides a realistic and complete 
estimate of what it would cost to rephcate the program in a similar setting. 

Operating the Future Steps program cost an estimated $333,214 per year (Table II. 8). 
Costs were incurred from program activities carried out by both SCC and the Ilhnois 
Department of Human Services (IDHS).® Overall, more than two-thirds (71 percent) of 
totd costs were labor costs, including wages and fringe benefits.^® One- tenth of total labor 
costs was SCC staff time that was uncompensated overtime or "donated" by student 
intems.^^ The other program costs mostly included supportive services made to participants 
(12 percent) and general administrative and other indirect costs (9 percent).^’ 



7 We selected October 1, 2001, to September 30, 2002, as the cost analysis period. Overall, the program 
operated during the two-year period from July 1, 2001, to September 30, 2001 

® We excluded costs explicitly related to participating in the evaluation. We estimate that any costs that 
could not be fully excluded were very small (that is, less than one percent). 

9 ID HS paid for the Future Steps program using Illinois state TANF funds. The costs of the Rural WtW 
Strategies Demonstration Evaluation were covered by the U.S. Department of Health and Human Services, 
Administration for Children and Families. 

10 To construct an estimate of the program's labor costs, we relied heavily on interviews with staff. 
D uring in-depth staff interviews conducted as part of our site visits, we asked staff to reflect on how they spent 
their time and used resources. D rawing on our understanding of Future Steps' processes, we engaged strff in 
detailed discussions about their time and activities. In particular, we asked them to estimate the amount of time 
they spent in a typical month (overall and on different program functions) and how their time varied 
throughout the month and cost period. Using the information they provided, along with data on staff wages, 
salaries, fringe benefits, and other relevant unit costs, we developed an estimate of total labor costs. The labor 
estimate for SCC is based on actual staff wage and salary data. In contrast, the labor estimate for ID HS staff is 
based on average salaries for administrators, managers, and case management staff. 

11 Overall, we estimated the value of uncompensated and donated time as $24,000. Nearly three- fifths of 
the amount represents support staff; most of the rest represents career speciahst time. 

12 For nonlabor costs, we relied mainly on accounting records. In addition to line-item expenditures, SCC 
provided an overhead rate that allowed us to value general administrative and other indirect costs. Except for 
travel costs, IDHS incurred no specific line-item expenses. Although IDHS did not use an organizational 
overhead rate, it was able to provide us with information on local lease costs that allowed us to estimate a 
comparable overhead rate for its general administrative and indirect costs. 
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Table 11.8. Total Estimated Costs for a One-Year Period of Future Steps Program Operations 
(In Dollars^ 



Type of Cost 


Shawnee 

Community 

College 

(SCC) 


Illinois 

Department of 
Human Services 
(IDHS) 


Total 


Labor Costs’’ 


Administrators 


7,893 


7,488 


15,381 


Managers 


56,993 


14,571 


71,564 


Case managers 


108,691 


12,388 


121,080 


Support staff 


28,079 


— 


28,079 


Subtotal — Labor Costs 


201 ,656 


34,447 


236,103 


(Percentage of totai) 


(72) 


(63) 


(71) 


Other Costs 


Supportive service payments 


39,476 


— 


39,476 


Generai student supportive services 


4,233 


— 


4,233 


Travei 


7,031 


505 


7,536 


Office suppiies 


1,797 


— 


1,797 


Audit services 


350 


— 


350 


Administrative and other indirect costs 


24,182 


4,998 


29,180 


Donated faciiities and suppiies 


— 


14,540 


14,540 


Subtotal — Other Costs 


77,068 


20,043 


97,111 


(Percentage of totai) 


(28) 


(37) 


(29) 


Total Costs for One- Year Period 


$278,724 


$54,490 


$333,214 


(SCC and IDHS percentage of total) 


(84) 


(16) 





®We estimated costs for the period October 1 , 2001 , to September 30, 2002. 

“^Labor costs inciude fringe benefits. The administrator costs inciude time spent by senior-ievei 
administrators within SCO and the state and regionai IDHS offices. The manager costs inciude time 
spent by the Future Steps director and coordinator (both SCO empioyees), as weii as the county- 
ievei IDHS supervisors. The case manager costs inciude time spent by the Future Steps career 
speciaiists (empioyed by SCO), as weii as the county-ievei IDHS case managers. 
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Nearly all program costs (84 percent) resulted from activities carried out by SCC (Table 
II.8). Most of SCO's costs were labor costs. The Future Steps career specialists alone 
comprised one-third (33 percent) of total program costs.^^ Their portion of costs was kept 
relatively low, given the program's general reliance on part-time career speciahsts— four of 
the five worked part-time and did not receive a fuU set of benefits. The work of the part- 
time Future Steps director and full-time coordinator also comprised a substantial fraction of 
total costs— close to one-fifth (17 percent).^^ 

IDHS also played a valuable role in operating the program. In particular, IDHS 
contributed to program planning and management and supported Future Steps staff in 
implementing the program locally. T en percent of total program costs represent IDHS staff 
time. Several administrators in the state ID HS office and two in the regional office helped 
develop program plans and oversee program implementation. County- level IDHS 

supervisors and fronthne staff coordinated with Future Steps career specialists regarding 
service delivery to shared clients.^^ In addition, IDHS donated office space and supphes to 
the program. The Future Steps career speciahsts were housed in the IDHS offices and used 
IDHS computers, telephones, and supplies. The value of this donated office space and 
supplies represents 4 percent of total program costs (or more than one-quarter of total 
ID HS costs— 27 percent).^® 

Reflecting differences in accounting methods, our estimate of total program costs was 
70 percent higher than the total expenditures tracked by the program for the same time 
period. The difference reflects our focus on assigning a market value to all resources used in 
operating the program, which went beyond a simple accounting of the expenses directly 
incurred or funded. First, the program only tracked expenditures incurred by SCC— it did 
not track the expenses incurred by ID HS, as we did. Second, the program did not track the 
labor cost of the SCC senior-level administrators. The time these staff spent providing 
administrative guidance and oversight to the program was paid for by SCC through its 
general administrative funds, rather than through project-specific funds. Third, project 
accounts undervalued the cost of the part-time project director's time. Because her other 
role as director of SCC's Placement Office overlapped somewhat with her Future Steps role, 
we counted a small fraction of her Placement Office time as a Future Steps cost.^^ Fourth, 

13 The Future Steps career specialists are categorized in T able II.8 as SCC case managers. 

14 The Future Steps director and coordinator are categorized in T able II.8 as SCC managers. 

In valuing the time spent by county- level IDHS staff, we did not consider the time that we assumed 
would have stUl occurred if the program did not exist. 

16 We developed the estimate of the donated office space and supplies based on information provided by 
IDHS on local lease costs, other indirect costs, and the number of sttff per county office. For each county 
office, we valued the cost of the space and supplies the Future Steps worker used in proportion to the value of 
her wages and benefits relative to the wages and benefits of all staff in the office. In so doing, we developed a 
conservative, lower-bound estimate of the value of the space and supplies. 

11 We counted nine percent of the director's Placement Office time as a Future Steps cost. We arrived at 
this fraction by considering the number of Future Steps participants in proportion to the number of SCC 
students and by assuming diat relatively more effort was required to place a Future Steps participant in a job 
compared with the average SCC student. 
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program accounts did not include SCO's general administrative and other indirect costs. 
Finally, SCO did not track volunteer or uncompensated staff time. 



• The average overall cost of serving a Future Steps participant was $2,90L 

Our estimate of total program costs, coupled with FSIS enrollment and service use data, 
formed the basis for assessing the average overall cost per participant. In our calculation of 
the cost per participant, we considered chents to be participants if they (1) enrolled during 
the cost period or the three months before the cost period ("cost period enroUees"), and 
(2) received at least one program service or contact during the cost period. Program 
participation for the cost period enroUees was nearly universal (98 percent). 

Future Steps chents participated over a relatively long period at a modest monthly cost. 
Cost period enroUees participated for an average of about 17 months, at a monthly cost of 
$170 and an overaU cost of $2,901 (Table II. 9). We used several steps to calculate this 
estimate. First, we calculated the average overaU duration of participation for the cost period 
enroUees.^®^ Second, we calculated the number of months during the cost period that each of 
the participants was enrolled in the program. We summed these months to count the "total 
person-months of participation" during the cost period for aU cost period enroUees. Third, 
we calculated the "average cost per-participant month" by dividing the total program cost 
estimate by the total person-months of participation. Fourth, we calculated the "average 
total cost per participant" by multiplying the average overaU duration of participation in the 
program by the average cost per-participant month. 

Future Steps cost less for the average participant than other recent welfare-to-work 
programs. Findings from the cost study of the Evaluation of the Welfare- to- Work (WtW) 



Table 11.9. Average Total Cost per Future Steps Participant 



Total 

Program 

Cost 

(One Year) 


Total 

Participants 
During 
Cost Period 


Total Person- 
Months Of 
Participation 
During Cost 
Period 


Average 
Overall 
Duration Of 
Participation 
(Months) 


Average Cost 
per-Participant 
Month® 


Average Total 
Cost per 
Participant 


$333,214 


266 


1,964 


17.1 


$170 


$2,901 



^Average cost per-participant month = (totai program cost / totai person-months of participation). 

‘’Average totai cost per-participant = (average cost per-participant month x average overaii 
duration of participation). 



We estimated the average overall duration of participation for cost period enroUees. As there was no 
program exit date, we estimated the duration of participation as the length of time between the date of a client's 
enrollment into the program and the date on which that client received his or her last program service. 
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Grants Program offers a useful point of comparison. Nine of the WtW programs in this 
national evduation implemented an "enhanced direct employment" model that was generally 
comparable to Future Steps. Like Future Steps, these programs operated outside of the 
welfare agency, focused on moving participants into employment fairly quickly, and 
provided a range of supportive services. On average. Future Steps cost close to $800 less 
per participant than these nine WtW programs— $2,901 versus $3,699 (Perez-Johnson et al. 
2002).^® The average cost of serving a Future Steps participant was likely lower than these 
other WtW programs for two key reasons. First, Future keps targeted a relatively broad 
segment of the WtW population, rather than focusing on hard-to-employ clients. As a 
result. Future Steps services were relatively less intensive and costly than many of the other 
WtW programs. For example, some of the other programs included subsidized work 
experience opportunities, which added to overall costs. Second, it is hkely that the 
administrative and other overhead costs were somewhat lower in rural Ilhnois than in the 
urban areas in which most of the other programs were located. 

• Future Steps did not spend all of Us available funding. 0 veiall, the program kept 

Us costs down by underspending on supportive service payments and st^ 

resources. 

The Future Steps program did not expend all of its available funding. The program- 
tracked expenditures for the one-year cost study represented about three-quarters of the 
program's one-year budget. That is, the program spent $70,000 less than its available 
funding for a one-year period of operations. 

The Future Steps program contained costs through smaller-than-expected investments 
in both supportive service payments and staff resources. First, supportive service payments 
were lower than projected. The program might have used supportive service payments more 
actively than it did to address clients' ongoing needs. Future Steps projected that it would 
spend approximately $88,000 per year on supportive service payments. During the cost 
period, the program spent $39,476— less than half the budgeted amount.^® As described 
earlier, some chents received no payments, and others received substantially less than the 
program hmit. Second, staffing costs were somewhat lower than anticipated. Although the 
program director was a full-time SCC employee, it was intended that she spend only part of 
her time (60 percent) on the Future Steps program. In the second half of the cost period, 
however, her available time for Future Steps fell below 50 percent, as she took on additional 
project responsibihties at SCC. In addition, the overall time spent by the career speciahsts 
was lower than expected due to staff turnover and resulting short-term gaps in 
caseload coverage. 



IS The estimates from the WtW Evaluation's cost study were converted from 2000 dollars into 2002 
doUans using the G ross D omestic Product chain- type price index. 

20 Across the two-year demonstration period, Future Steps provided $93,017 in supportive service 
payments to participants. This represented 53 percent of the totd amount budgeted for diese payments 
($176,000 across two yeans). 
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Impacts on Employment, 

SE L E - Su E E I C I E N C Y , AND 

Well-Being 



T hrough its employment-focused, case management model, Future Steps aimed to 
help chents develop job-related skills, enhance practical hfe skills, overcome logistical 
and personal challenges, and make a successful transition into the workforce. The 
services and support dehvered through the program were intended to help chents maintain 
employment, advance in the workforce, improve well-being, and, ultimately, achieve 
economic independence. We expected that, 18 months after they enrohed in Future Steps, 
the program group members, on average, would have significantly higher employment rates 
and earnings, significantly lower TANF and family poverty rates, and greater levels of 
personal and family weh-being than their control group counterparts. 

Overah, Future Steps program group members received substantially more services than 
control group members, but there was no evidence that Future Steps led to improved 
employment and earnings or that it reduced welfare dependence or personal and family 
hardships. In this chapter, we examine findings related to sample members' use of services 
and the effects of the Future Steps services on their employment, self-sufficiency, and well- 
being. We first compare the types of services and resources that both program and control 
group members used. We then examine program impacts on employment, public assistance, 
and personal and family well-being. Finally, we assess whether Future Steps led to 
differences in outcomes for key subgroups. In Chapter IV, we more fully examine the 
factors that explain the findings and discuss the implications of the findings for designing, 
implementing, and testing welfare-to-work programs in rural areas. 

The following discussion of program impacts focuses on the 18-month period after 
sample members were randomly assigned into the Future Steps program or the control 
group. We draw primarily on data from the 18-month chent survey and, where appropriate, 
we integrate findings based on administrative records data. Chapter I contains a discussion 
of the analytic methods used for the impact study. Appendix C contains supplemental tables 
and figures. 
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Differencesin the Use oe Servicesand Resources 

For Future Steps to positively affect clients' employment, self-sufficiency, and well- 
being, program group members had to receive more services than control group members. 
In Chapter II, based on an analysis of program data recorded in the Future Steps 
Information System (FSIS), we observed that many program group members received a 
substantial amount of individualized support from Future Steps over an extended period. In 
this section, based on an analysis of 18-month survey data, we compare all of the services 
and resources used by program and control group members during the 18-month follow-up 
period, including those related to employment, education and training, and logistical and 
personal support. Though only program group members benefited from Future Steps, both 
groups might have received services from a wide variety of sources. G iven the intensity of 
Future Steps and the addition the program represented to the local service capacity, we 
expected that a higher fraction of program group members than control group members 
would receive different types of services. In the 18-month survey, we asked both groups the 
same set of guestions about the services they received. 

• SigniGcantly more program group members than control group members 

received employment-related assistance. 

Program group members were significantly more likely than control group members to 
report having received employment-related assistance. Future Steps emphasized providing 
participants with job search advice and individuahzed assistance to secure a job. To a lesser 
extent, the program worked with participants to enhance their general employability and 
readiness for work (for example, by coaching and mentoring them on workplace readiness 
issues and life skills). At the 18-month foUowup, nearly half of all program group members 
reported that they had received some type of employment preparation assistance during the 
18-month period after they enrolled in Future Steps (47 percent, compared with 28 percent 
of control group members) (Figure III.l). 

Job search and job placement assistance was the most common type of employment 
preparation service received. More than one-third (38 percent) of program group members, 
compared with one-fifth of control group members, reported that they had received some 
type of job search or job placement service, such as help writing a resume or finding a job 
opening, getting a better job, or arranging a job with an employer (Figure III.l). Although 
less prevalent, more than one-fifth (22 percent) of program group members, compared with 
16 percent of control group members, received some type of job readiness training to help 
prepare them for work. This training included mentoring on such topics as getting along 
with people at work, dressing for a job, and maintaining a work schedule. Significantly more 
program group members also received counsehng, help, or encouragement from a 
caseworker, job coach, or counselor about issues at work (11 versus 4 percent). Finally, 
Future Steps did not emphasize participation in work experience and on-the-job training 
opportunities, and about one-tenth of both program and control group members held work 
experience or on-the-job training positions during the foUow-up period (not shown). 
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Figure III.1. Employment and Training Services Received During the 18-Month 
Follow-Up Period 



Percentage 
50 47*** 




Any Job Search/ Job Readiness Work-Related Any Education 

Employment- Job Placement Training Counseling or Training 

Related Assistance 

Assistance 

■ Program Group □ Control Group 



Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica 
Policy Research, Inc. 

Note: Estimates were adjusted using multivariate regression methods, and data were weighted to 

account for survey nonresponse. Standard errors of the estimates account for sample weights. 

•/**/*** Significantly different from zero at the .05/.025/.01 level, one-tailed test. 



The duration and intensity of employment assistance was significantly higher among 
program group members than among control group members (not shown). Among sample 
members who received assistance, the average time period over which the assistance was 
received was twice as long for program group members as for control group members (eight 
versus four months). In addition, among those who received any type of employment- 
related assistance, program group members reported receiving help more times— 35 sessions 
or meetings, on average, compared with 20 for control group members. 

The fraction of program group members who reported on the survey that they had 
received employment services was substantially lower than the reports based on the 
program's FSIS database. As discussed in Chapter II, more than four-fifths of program 
group members received some type of employment service, based on FSIS data, compared 
with just under half, based on survey data. Since the survey was collected 18 months after 
program enrollment and, presumably, several months after most clients received services, 
some may not have recalled receiving them or may have perceived the service somewhat 
differently than it was characterized on the survey. However, the difference in service use 
between the two data sources also suggests that the services some chents received may have 
been relatively limited, or not intensive or distinctive enough to be recalled. 
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• About one-thud of all sample members participated in education and training 
during the 18-month follow-up period. Differences between program and control 
group members were not significant 

Many sample members, program and control group members alike, sought to improve 
their economic prospects through education and training. About one-third— 36 percent of 
program group members and 31 percent of control group members— reported participating 
in some type of education or training during the 18-month foUow-up period (Figure III.l). 
The difference was not significant, suggesting that Future Steps played a limited role in 
helping connect chents with local education and training opportunities. Among those who 
participated in education and training, the duration and intensity of training were similar for 
program and control group members. Both groups participated about 20 hours per week, 
and the average duration of training was about six months (seven for program group 
members and five for control group members). 

Vocational programs and course work were the most popular educational options for 
sample members. One-quarter of program and one- fifth of control group members 
participated in vocational education or training, which represented postsecondaiy course 
work for many. (This difference was not significant.) (See Appendix C, Figure C.l.) 
Computer-related programs and the Certified Nursing Assistant program were popular 
training options. Among those who participated in vocational training, about one-third of 
both groups earned a degree or certificate during the 18-month foUow-up period. Smaller 
fractions— less than one-tenth of both groups— took high school or GED classes or 
participated in adult basic education. Among chents who did not have a high school 
credential at basehne, about one-fifth of both groups had earned one by the time of the 
follow-up survey. 

• Program group members were much more likely than their control group 
counterparts to receive logistical support including help finding and paying for 
transportation, child care, and job-related clothing, tools, and supplies. 

Future Steps played an important role in helping clients access and pay for child care, 
transportation, and job-related clothing, tools, and supplies. These logistical issues can be a 
challenge for many low-income workers. About one-third of program group members, 
compared with a significantly smaller fraction of control group members, reported that they 
had received help paying for child care from the welfare agency or another agency; similar 
fractions of program group members reported receiving help paying for transportation and 
job-related materials (Figure III. 2). These fractions were significantly higher than those for 
control group members. In particular, about one-quarter of program group members 
reported receiving gas vouchers, about one-tenth reported receiving money for car repairs, 
and another one-tenth reported receiving money for car registration, hcensing, or insurance 
(Appendix C, Table C.l). In addition to paying for logistical services, program group 
members were also significantly more likely to report that they had received help obtaining 
or making arrangements for child care, transportation, or work-related materials (Appendix 
C, Figure C. 2). 
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Figure III.2. Logistical Supportive Services Received During the 18-Month 
Follow-Up Period 

Percentage 




Help Paying for Child Care Help Paying for T ransportation Help Paying for Job-Related 

Clothing, Tools, or Supplies 



■ Program Group □ Control Group 



Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica 
Policy Research, Inc. 

Note: Estimates were adjusted using multivariate regression methods, and data were weighted to 

account for survey nonresponse. Standard errors of the estimates account for sample weights. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-tailed test. 



• Future Steps helped many clients with a wide range of personal and family 
needs. A small hut significantly higher fraction of program group members, 
compared with control group members, received life skills training and 
mediation services. 

As part of Future Steps case management services, career specialists helped many clients 
with personal and family issues. Differences between the program and control group 
members were generally not significant however, likely reflecting the program's lesser focus 
on personal and family assistance, compared with employment-related assistance. More than 
two-fifths of both groups reported receiving some tj^e of personal or family service during 
the 18-month follow-up period (47 versus 42 percent respectively) (Table III.l). About 
one-guarter of both groups received a health-related service, though the fraction that 
reported receiving services for mental health and medical attention for physical conditions 
was much lower than the fraction that were identified as facing mental and physical health 
challenges. (The prevalence of obstacles and hardships is discussed later in this chapter.) 
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Table III.1. Personal and Family Services and Assistance Received During the 18-Month 



Follow-Up Period 



Type of Service (Percentage) 


Program 

Group 


Control 

Group 


Estimated 

Difference 


Any Personal or Family Service 


47.1 


42.3 


4.9 


Life Skills Training 


13.7 


8.5 


5.2* 


Personal Encouragement 


Counseling/encouragement on personal Issues 


12.7 


10.5 


2.2 


Support or discussion groups 


7.3 


7.8 


-0.6 


Advocacy 


Legal assistance 


10.6 


13.7 


-3.1 


Mediation with employers, landlords, others 


7.8 


1.7 


6.0*** 


Help finding housing 


4.7 


5.1 


-0.4 


Health-Related Services 


Any health-related service 


24.4 


25.5 


-1.1 


Mental health services or counseling 


9.9 


9.0 


0.9 


Substance abuse services or treatment 


2.5 


2.6 


-0.1 


Domestic violence counseling 


4.2 


3.4 


0.8 


Medical attention for physical condition 


12.2 


14.2 


-2.1 


Household member received counseling related to 
mental health, substance abuse, or domestic 


violence 


7.9 


8.0 


-0.1 


Sample Size 


252 


268 





Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy 

Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The data were 

weighted to account for Interview nonresponse. Standard errors of the estimates 
account for sample weights. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-talled test. 



Significantly higher fractions of program group members than control group members 
received hfe skills training and mediation services (Table III.l). Life skills training in Future 
Steps focused on managing life responsibilities while working— for example, by managing 
money better, developing parenting and relationship skills, and balancing job and family 
responsibilities. Although a small fraction of program group chents (14 percent) reported 
receiving training on life skill issues during the follow-up period, an even smaller fraction of 
the control group (9 percent) was able to take advantage of such services. Future Steps 
career speciahsts also played an important role mediating employment and other issues for a 
small but significant fraction of program group members. Eight percent of program group 
members, compared with two percent of control group members, reported that a 
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caseworker, job coach, or counselor had helped them by talking to someone on their behalf 
(such as an employer or landlord) to help work out problems. 

Impacts ON Employment and Earnings 

Future Steps was designed to increase chents' employment and earnings through its 
employment-focused case management services. The services and support provided through 
the program were not only intended to help chents obtain a good job, but also to help them 
maintain employment, advance in the worlrforce, and increase their earnings. In this section, 
we present the impacts of the program on employment and earnings during the 18 months 
after sample members were randomly assigned either to the program or control group. We 
also highhght differences in the job characteristics of those sample members who were 
employed during the fohow-up period. 

• Overall, Future Steps did not improve clients' employment status, despite 

somewhat higher employment rates for program group memhers during the early 

months afler random assignment 

The employment rates of program and control group members were generahy similar 
during the 18-month follow-up period. Three- guarters of both groups were employed at 
some point during the fohow-up period, and a little more than half of both groups were 
employed 18 months after random assignment (Table III. 2). Both program and control 
groups experienced an overah increase in employment of roughly 20 percentage points 
during the follow-up period (Figure III. 3). This increase partly reflects the work orientation 
of the people the program targeted. At basehne, ah sample members stated that they were 
available to work 30 hours per week. Many also were reguired to do so to receive TANF or 
food stamp benefits. 

Although program group members experienced somewhat higher employment rates 
than control group members during the early months after random assignment, these 
differences did not persist (Figure III.3; Appendix C, Tables C.2 and C.3). Given the job 
search and job placement assistance the program provided, the early gains in employment 
for the program group were not unexpected. Based on survey data that tracked employment 
month by month, there was a significant difference in employment only during the fourth 
month after random assignment (48 versus 40 percent for the program and control groups, 
respectively). Moreover, the findings based on administrative records data show that 
program group members were employed at significantly higher rates during the first three 
guarters after random assignment (seven to nine percentage points higher than control 
group members per guarter) (Appendix Table C.3).^ These early employment differences, 
however, did not persist throughout the follow-up period based on either the survey or 
administrative data. 



1 We examined employment and earnings using both monthly survey data and quarterly administrative 
records data. While the discussion here focuses on the survey data analyses, some of the tables in Appendix C 
show results based on administrative data. 
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Table III.2. Employment and Earnings During the 18-Month Follow-Up Period 





Program 


Control 


Estimated 


Outcome 


Group 


Group 


Impact 


Employment (Percentages) 








Employed 18 months after random 
assignment 


54.2 


54.9 


-0.6 


Ever employed during follow-up period 


74.5 


75.1 


-0.6 


Months employed during follow-up period 
(percentage) 








0 


24.6 


21.3 


3.3 


>0 to 24 


8.2 


13.6 


-5.4 


25 to 49 


17.9 


15.3 


2.6 


50 to 74 


16.4 


15.3 


1.1 


75 to 100 


33.0 


34.6 


-1.5 


Average 


48.7 


49.1 


-0.5 


Number of jobs during follow-up period 








0 


23.6 


20.6 


3.0 


1 


32.0 


46.2 


-14.2*** 


2 or more 


44.4 


33.2 


2*** 


Average number of jobs 


1.5 


1.3 


0.2** 


Earnings (Dollars) 








Monthly earnings during follow-up period 
(average for all clients) 


568 


614 


-45 


Total earnings during follow-up period 
(average for all clients) 


8,831 


9,381 


-550 


Sample Size 


252 


268 





Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy 

Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The data were 

weighted to account for Interview nonresponse. Standard errors of the estimates 
account for sample weights. 

7*7*** Impact estimates are statistically significant at the .05/.025/.01 level, one-talled test. 
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Figure III.3. Employment Rates During the 18-Month Follow-Up Period, by Month 




“ Program Group —Control Group 



Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica 
Policy Research, Inc. 

Note: Estimates were adjusted using multivariate regression methods, and data were weighted to 

account for survey nonresponse. Standard errors of the estimates account for sample weights. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-talled test. 



Program and control group members both cited a wide range of reasons for not 
working. Among those not working at the time of the 18-month survey, the most common 
reasons for not working were transportation problems (22 percent), health problems of their 
ovm or a family member (21 percent), child care problems (8 percent), participation in 
school or training (6 percent), or a recent firing or layoff from a job (5 percent) (not shovm). 
Overall, these reasons were similar for program and control group members. Logistic 
regression analyses also show that chents who lacked a high school credential or had limited 
work experience were more hkely than others to be unemployed at followup. Younger 
clients also were more likely than others to have not worked at all during the follow-up 
period. 

• Future Steps had no impact on sample members' earnings. 

Earnings were similar for program and control group members during the 18-month 
follow-up period. The average program group member earned $568 per month, compared 
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with $614 for the average control group member (Table III. 2; Appendix C, Table C.4).^-^ 
Findings were comparable, and also insignificant, based on administrative records data 
(Appendix C, Table C.3).^ Like employment rates, the pattern of average monthly earnings 
for both groups rose during the follow-up period, particularly during the first seven months 
(Figure III. 4). The earnings increase partly reflects sample members' rising employment 
over the same period. It also reflects the greater likehhood that those with relatively high 
wages or long hours (or both) would stay employed and that some of those who remained 
employed would experience an increase in earnings. 

Total earnings were low for both groups. Across the 18-month period, average total 
earnings egualed roughly $9,000 for both groups (Table III. 2). (Overall earnings among 
those who were employed at some point during the follow-up period were still low— about 
$19,000, on average, for the 18-month period.) Overall, the low earnings likely reflect the 
limited skills of many sample members, along with the relatively hmited employment 
opportunities in southern Illinois. 

• Job turnover was common, and program group members beld more jobs than 
control group members. However, the job changes the program group made did 
not reflect a move into better or higher-paying jobs. 

Program group members held more jobs than their control group counterparts during 
the 18-month foUow-up period, and this difference was statistically significant. This may 
reflect more freguent transitions between jobs for the program group. More than two-fifths 
(44 percent) of program group members, compared with one-third of control group 
members, held two or more jobs during the follow-up period (Table III. 2). Moreover, 
working program group members held their current or recent job for less time, on average, 
than their control group counterparts— 12 months versus nearly 15 months (Table III.3). 
These findings may partly reflect the better, more freguent job placement information to 
which program group members had access.^ 



2 Earnings for each of the months during the foUow-up period were constructed based on information 
collected through the survey on job start and stop dates, the usual number of hours and days worked per week, 
and the hourly wage rate. 

2 The average amount includes all sample members, whether or not they had earnings during the follow- 
up period. More than one- fifth (22 percent) of the sample earned $0 during the follow-up period. D uring the 
18th month after random assignment, 46 percent of the sample earned $0. 

4 When converted into average monthly earnings, the average quarterly earnings based on administrative 
records data were, not unexpectedly, substantially lower for both program and control group members than the 
estimated earnings based on survey data. Although the reported administrative wage data represent accurate 
information on earnings reported by employers, the reporting is not comprehensive, as the administrative data 
exclude out-of-state earnings, self-employment, and informal jobs that are unreported. 

^ The Future Steps program may have had either a direct or indirect influence on helping sample 
members learn about and obtain jobs. However, program and control group members reported using similar 
methods to find jobs. About two-fifths of both groups (38 percent of program and 43 percent of control 
group members) learned about their current or most recent job by contacting the employer directly, filling out 
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Figure III.4. Average Earnings During the 18-Month Foiiow-Up Period, by Month 




Month 



“ Program Group —Control Group 



Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica 
Policy Research, Inc. 

Note: Estimates were adjusted using multivariate regression methods, and data were weighted to 

account for survey nonresponse. Standard errors of the estimates account for sample weights. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-talled test. 



The job changes that program group members made neither reflected a move into 
better jobs nor translated into wage or job advancement gains. As described below, wages 
during the follow-up period were comparable for program and control group members, as 
were the fractions of employed program and control group members who received fringe 
benefits through their jobs (Table III.3). Moreover, about one-tenth of both program and 
control group members (11 and 9 percent, respectively) reported either being promoted to a 
position with higher pay and greater responsibihties at their current or most recent employer, 
or thinking that they were likely to be promoted to such a position in the next year 
(not shown). 



(continued) 

applications, or sending in resumes. Similarly, two-fifths (40 percent) of program group members, compared 
with 34 percent of control group members, learned about their job through a friend or relative. A small 
fraction of both groups (eight percent of program group members and five percent of control group members) 
reported learning about their current or most recent job through a welfare or employment agency or a school- 
based employment center. N one of these program versus control differences was significant. 
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Table III.3. Characteristics of the Current or Most Recent Job, for Sample Members 
Who Were Employed During the Follow-Up Period 


Outcome (Percentages)® 


Program Group 


Control Group 


Hourly Wage Rate 


Less than $5.00 


9.8 


9.8 


$5.00 to $6.99 


51.7 


42.1 


$7.00 to $8.99 


27.2 


31.3 


$9.00 or more 


11.3 


16.8 


(Average hourly wage) 


$6.78 


$6.96 


Usual Hours Worked per Week 


Less than 20 


7.5 


13.1 


20 to 29 


16.6 


13.7 


30 to 34 


12.7 


14.4 


35 to 44 


47.4 


46.6 


45 or more 


15.8 


12.3 


(Average hours per week) 


36.1 


34.9 


Monthly Earnings 


Less than $600 


18.8 


20.1 


$600 to $999 


33.1 


30.5 


$1,000 to $1,399 


27.5 


29.8 


$1,400 to $1,799 


12.1 


13.3 


$1 ,800 or more 


8.6 


6.4 


(Average monthly earnings) 


$1,082 


$1,056 


Fringe Benefits 


Health Insurance 


48.7 


45.5 


Paid sick leave 


23.8 


27.9 


Paid vacation 


41.6 


42.2 


Paid holidays 


42.9 


45.5 


Dental benefits 


28.6 


31.0 


Retirement or pension benefits 


25.3 


32.6 


Number of Months on Job 


1 to 3 


22.7 


21.6 


4 to 6 


19.9 


17.8 


7 to 12 


27.0 


20.8** 


13 to 18 


17.7 


14.2 


More than 18 months 


12.7 


25.6** 


(Average months) 


12.2 


14.8 


Sample Size 


193 


212 



Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica 

Policy Research, Inc. 

Note: The data were weighted to account for Interview nonresponse. Standard errors of 

the estimates account for sample weights. 

®About one-quarter of both the program and control groups did not work during the follow-up 
period. Since these cases are not Included In this table, we do not report estimated Impacts 
for these outcomes. However, we do report statistically significant differences between the 
two groups: 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-talled test. 
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• Both program and control group members held similar, low-wage jobs. Their 
jobs were low-paying compared with other low-income workers nationally and in 
the state of Illinois. 

Wages are one of the strongest indicators of job guality. Among employed program 
group members, the average wage in their current or most recent job was $6.78, which was 
comparable to the $6.96 earned by control group members (Table III. 3). The average 
program group member earned $1,082 per month; earnings for control group members were 
similar (Table III. 3). Future Steps clients earned relatively low wages compared with other 
groups of low- wage workers nationally and in the state of Illinois. A study of welfare leavers 
in eight states showed that the average hourly wages for welfare recipients about 12 months 
after leaving welfare ranged from $7.95 to $9.26, with an average wage of $8.36 in Illinois 
(Acs and Loprest 2001).® This disparity likely reflects the limited employment opportunities 
available in southern, rural Illinois. 

Program and control group members held comparable jobs (Table III. 3; Appendix C, 
Table C.5). Most worked in service jobs (such as in the h^th and food sectors) or held 
production or trade-related positions. The average program group member worked 36 
hours per week in his or her current or most recent job. The jobs they held offered 
relatively few fringe benefits. For instance, only half of working program group members 
were in jobs that offered health insurance, and about one-guarter held jobs that offered paid 
sick leave. These fractions were similar for control group members. Nearly two-fifths of 
both program and control group members worked evenings or nights or in jobs with 
variable, rotating, or irregular shifts. D ependable child care can be more difficiit to find 
during these nonstandard work hours. 

Changes IN Welfare Dependence and Self-Sufficiency 

Future Steps was designed to reduce welfare dependence and improve the self- 
sufficiency of its clients by supporting their efforts toward employment and ongoing labor 
market success, and helping them address challenges in their lives. In this section, we 
summarize program impacts on public assistance, income, and self-sufficiency during the 
18-month period after clients were randomly assigned to the Future Steps program or the 
control group. 

• A year and a half after random assignment, the program had not led to 
significant reductions in the receipt of TANF, food stamps, or other forms of 
public assistance. 

Similar fractions of program and control group members reported receiving TANF, 
food stamps, and other t^es of public assistance during the month before the 18-month 



6 We converted the average hourly wage estimates from Acs and Loprest (2001) into 2003 dollars (from 
2000 dollars). The majority of sample members in the Future Steps evaluation reported wages in 2003 dollars. 
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follow-up survey (Table III.4). More than 8 in 10 sample members received some form of 
public assistance during the month before the survey. Relatively few received TANF, 
however: on the survey, only about one in seven of both program and control group 
members reported that they had received TANF during the previous month. A much higher 
fraction— about three in four of both groups— reported receiving food stamps during the 
month before the survey. Small fractions received other types of pubhc assistance— for 
example, about one in eight received Supplemental Security Income or Disability Insurance. 

We also examined TANF and food stamp receipt throughout the 18-month follow-up 
period. From the start to the end of the follow-up period, the fraction of sample members 
who received TANF and food stamps changed only modestly, with comparable overall rates 
of dechne for the two groups. For example, the fraction that reported that they had received 
TANF during the month before the 18-month survey was ^out two percentage points 
lower than at baseline for both program and control group members (not shovm). The 
survey findings for food stamps, as well as findings for TANF and food stamps based on 
administrative records data, were generally eguivalent and followed a comparable pattern 
(Appendix C, Tables C.6 and C.7). Using the administrative data, we also examined the 
fraction of sample members who received TANF and food stamps at any point during the 18- 
month follow-up period, finding no significant reductions in the fraction of program group 
members who depended on public assistance (Appendix C, Table C.8).^ 

In addition to TANF and food stamps, we examined whether the fraction that claimed 
the federal Earned Income Tax Credit (EITC) was different for the program and control 
groups. The EITC is an important source of assistance for working famihes. Given that 
Future Steps advised clients on a vUde range of economic and personal issues, we wanted to 
explore whether a higher fraction of program group members had received the EITC. 
Through an analysis based on several survey guestions, we estimated that a similarly large 
fraction of both groups had received, or were likely to have received, the EITC during the 
most recent tax year (70 percent of program group members, compared mth 67 percent of 
control group members) (not shovm).® According to our estimates, among program and 
control group members who did not receive the EITC, about one- third of both groups were 
likely to have been ehgible for it. 



’ About one-third (32 percent) of program group members, compared with 28 percent of control group 
members, received TANF at some point during the 18-month follow-up period (Appendix C, T able C.8). This 
difference was not significant. 

® We considered a sample member likely to have received the federal EITC if they reported receiving or 
applying to receive it or if three conditions were met: (1) the estimated annual household earnings of the 
sample member were below the EITC limit, factoring in differences by family size; (2) someone else had 
prepared the sample member's tax return; and (3) the sample member had received a federal refund. 
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Table III.4. Participation Rates in TANF, Food Stamps, and Other Public Assistance 
Programs During the Month Before the 18-Month Survey 



Outcome (Percentage Received) ® 


Program 

Group 


Control 

Group 


Estimated 

Impact 


Temporary Assistance for Needy Families 
(TANF) 


14.3 


13.1 


1.3 


Food Stamps 


75.6 


71.5 


4.0 


Women, Infants, and Children (WIC) 


29.6 


28.0 


1.6 


Supplemental Security Income or Disability 
Insurance 


13.8 


13.3 


0.5 


Social Security 


3.7 


6.4 


-2.7 


Unemployment Insurance 


5.9 


4.8 


1.1 


General Assistance 


2.7 


2.3 


0.3 


Other Public Assistance 


1.3 


1.4 


-0.1 


Any Public Assistance (Any of the Above)*^ 


84.2 


82.1 


2.2 


Sampie Size 


252 


268 





Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy 

Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The data were 

weighted to account for Interview nonresponse. Standard errors of the estimates 
account for sample weights. 

®The outcome measures represent the percentage of sample members whose household received 
the benefit during the month before the 18-month follow-up survey. The month before the survey 
represented a different number of months after random assignment for different clients. For 
example, for some clients, the month before the survey represented 18 months after random 
assignment. For others. It represented from 19 to 23 months after random assignment. 

“^Receipt of foster care assistance Is also represented In this aggregate category. However, the 
point estimates for the receipt of foster care assistance were too small to report. 

7*7*** Impact estimates are statistically significant at the .05/.025/.01 level, one-talled test. 
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• The program had no effect on household income or poverty. At the 18-month 

followup, more than two-thirds of program group members hved in households 
whose monthly income was helow the federal poverty level 

On average, program group members had household income totaling $1,244 during the 
month before the survey. This amount was not significantly different from the control 
group's total household income ($1,285) (Figure III. 5). Our es t imate of household income 
is derived from three primaiy sources of income during the month before the survey: 
(1) chent's own earnings; (2) other private income sources (primarily, earnings of other adults 
in the household); and (3) pubhc assistance (primarily, food stamps and TANF). 

Program and control group members each relied on three main sources of income: 
their earnings, food stamps, and earnings from another adult in the household (Figure III.5, 
Table III. 5). On average, sample members' own earnings represented only about one-third 
of total income during the month before the survey. (About half of all sample members had 
their own earnings at that time [Appendix C, T able C.9]). The largest component of income 
was pubhc assistance. Weh over two-fifths of the average monthly income was derived from 
different forms of pubhc assistance, with the value of food stamps representing more than 
one-guarter for both program and control group members. Finahy, nearly one-guarter of 
average monthly income represented earnings from other private income sources, such as 
earnings of a spouse, partner, or another adult in the household. About one in seven sample 



Figure III.5. Impacts on Average Monthly Income from Earnings and Other Sources 
(by Household During the Month Before Survey) 



Average Monthly Income 




Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica 
Policy Research, Inc. 

Note: Estimates were adjusted using multivariate regression methods, and data were weighted to account for survey 

nonresponse. Standard errors of the estimates account for sample weights. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-tailed test. 
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Table III.5. Impacts on Monthly Income Sources 





Percentage of Total Income from Source® 




Program 


Control 


Estimated 


Outcome 


Group 


Group 


Impact 


Own Earnings 


31.5 


34.4 


-2.8 


Other Private income Sources 


22.0 


23.2 


-1.1 


Spouse or Partner’s Earnings 


7.3 


7.5 


-0.2 


Other Household Members’ Earnings 


9.7 


10.5 


-0.8 


Earnings from Informal/Odd Jobs‘s 


1.5 


1.4 


0.1 


Child Support 


1.7 


1.8 


-0.1 


Other Private Income Sources 


1.8 


2.0 


-0.2 



Totai Pubiic Assistance 


46.5 


42.5 


4.0 


TANF 


4.8 


3.3 


1.4 


Food Stamps 


28.8 


27.7 


1.2 


WIC 


2.4 


1.7 


0.7 


SSI 


6.0 


4.7 


1.3 


Social Security 


0.7 


1.6 


-0.9 


Unemployment Insurance 


2.9 


1.9 


1.0 


General Assistance 


0.8 


0.8 


0.0 


Foster Care 


0.0 


0.3 


-0.3 


Other Governmental Assistance 


0.1 


0.5 


-0.4 



Totai income (Aii Sources) 


100.0 


100.0 


Sampie Size 


245 


259 



Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy 

Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The data were 

weighted to account for Interview nonresponse. Standard errors of the estimates 
account for sample weights. 

®By household, during the month before the 18-month follow-up survey. The month before the 
survey represented a different number of months after random assignment for different clients. For 
example, for some clients, the month before the survey represented 18 months after random 
assignment. For others. It represented from 19 to 23 months after random assignment. 

“^Earnings from Informal or odd jobs may have been jobs held by either the sample member or 
another adult household member. 

7*7*** Impact estimates are statistically significant at the .05/.025/.01 level, one-talled test. 
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members reported earnings from a spouse or partner during the month before the survey, 
while about one in six reported earnings from another adult in their household, such as a 
parent, sibhng, or friend (Table C.9).® 

Overall, two-thirds of program group members were poor at the time of the survey. 
That is, two-thirds had household income during the month before the survey that was 
below the federal poverty guidehnes for the size of their household (Figure 111.6).^“ Many 
were in extremely poor households: 3 in 10 program group members had household income 
less than 50 percent of the poverty level. The overall poverty rate was a bit lower for control 
group members, though the difference was not statistically significant. 



Figure III.6. Households Living Above and Below Poverty at 18-Month Followup 



Percentage 



- 


32 




38 


38 


34 


- 


30 




28 


Program Group Control Group 



□ 1 00% of Poverty or More 

□ Greater than 50% of Poverty and Less than 100% 
■ Below 50% of Poverty 



Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica 
Policy Research, Inc. 

Note: Estimates were adjusted using multivariate regression methods, and data were weighted to 

account for survey nonresponse. Standard errors of the estimates account for sample weights. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-tailed test. 



9 There were no significant differences in the household composition of program and control group 
members at either baseline or foUowup. However, at the time of the 18-month foUowup, a smaller fraction of 
both program and control group members reported that they were living as a single-adult household. This may 
partly reflect the downturn in the local economy during the foUow-up period. At followup, 42 percent of all 
sample members were living as a single-adult household, 31 percent with a spouse or partner, and 27 percent 
with another adult. 

The poverty levels we report are based on U.S. Department of Health and Human Services federal 
poverty guidelines for year 2003. For instance, based on these guidelines, a family of three is considered to be 
in poverty if its monthly income is below $1,272 ($15,260 on an annual basis), and a family of four is poor if its 
monthly income is below $1,533 ($18,400 on an annual basis). 
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Impacts ON Well-Being and Quality oe Liee 

Future Steps was intended to improve clients' life skills, reduce challenges they faced, 
and increase their personal well-being. Program designers anticipated that these short-term 
outcomes would, in turn, prepare chents to get and keep jobs, advance in employment, and 
make progress toward self-sufficiency. In this section, we examine impacts on personal and 
family functioning and guality-of-life outcomes. 

• In terms of personal functioning, program group members scored significantly 
higher than control group members on scales that measured self-efficacy and 
orientation toward the future. 

To assess differences in the personal functioning and attitudes of program and control 
group members at the time of the 18-month foUowup, we asked a series of guestions to 
measure self-efficacy, self-esteem, and future orientation (Table III.6). We used the Pearlin 
Mastery Scale to measure self-efficacy, or individuals' sense of their ability to control their 
life and manage the responsibihties, challenges, and opportunities that are a part of it (Pearlin 
and Schooler 1978). We used three guestions drawn from the Rosenberg Self-Esteem Scale 
to examine self-esteem (Rosenberg 1989). Finally, to examine sample members' outlook 
toward the future (their "future orientation") we pooled responses to three guestions. 
Overall, sample members' scores on these personal functioning scales were guite positive, 
with the average response to nearly aU of the guestions in the most positive guartUe. 

Program group members were significantly more likely than their control group 
counterparts to respond positively on the self-efficacy and future orientation guestions, 
suggesting that the program did help strengthen clients' motivation toward the future, as well 
as confidence about their abihties to achieve their goals. In measuring self-efficacy, for 
example, program group members were significantly more hkely to agree that "I can do 
anything I set my mind to" and significantly more likely to disagree that "There is no way I 
can solve some of the problems I have" (Table III. 6). Similarly, program group members 
were more hkely than control group members to agree that "I have a plan for the future" 
and that "I am confident that I will be able to reach my goals." These findings were 
consistent with reports from many focus group participants (discussed in Chapter II) that 
Future Steps had helped to strengthen their motivation and self-confidence and to give them 
a sense of control over their lives. Although Future Steps had no effect on employment, 
earnings, and welfare dependence, these findings show that the program was successful at 
helping participants feel good about their personal abilities and efforts. 

• Hardships were similarly pervasive for program and control group members. 

During the same time that sample members were preparing for work, getting jobs, and 
building labor market experience, most also faced substantial obstacles and hardships. Half 
of both program and control group members faced three or more serious personal or 
logistical challenges during the follow-up period (Figure III.7). These challenges can make it 
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Table III.6. Clients’ Self-Efficacy, Self-Esteem, and Future Orientation at the Time of the 18-Month 



Follow-Up Survey 



Characteristic® 


Program 

Group 


Control 

Group 


Estimated 

Impact 


Self-Efficacy or Sense of Control (Out of 28) 


22.1 


21.4 


0.6** 


There Is no way 1 can solve some of the problems 1 have 


3.0 


2.8 


0.2** 


1 feel that 1 am being pushed around In life 


3.0 


3.0 


0.0 


1 have little control over the things that happen to me 


3.1 


3.0 


0.1** 


1 can do anything 1 set my mind to 


3.4 


3.3 


0.1*** 


1 feel helpless In dealing with the problems In my life 


3.1 


3.0 


0.0 


What happens to me In the future depends on me 


3.4 


3.4 


0.1 


There Is little 1 can do to change the Important things In my life 


3.1 


3.1 


0.0 


Self-Esteem (Out of 12) 


9.5 


9.4 


0.1 


1 am able to do things as well as most people 


3.3 


3.2 


0.1 


1 certainly feel useless at times 


2.8 


2.8 


0.0 


All In all, 1 am Inclined to feel that 1 am a failure 


3.3 


3.3 


0.0 


Future Orientation (Out of 12) 


10.1 


9.8 


0.3** 


1 have a plan for the future 


3.3 


3.2 


0.1* 


1 am confident that 1 will be able to reach my goals 


3.3 


3.2 


0.1* 


1 feel 1 am responsible for my future and my chlld(ren)'s future 


3.5 


3.4 


0.1* 


Sample Size 


252 


268 


520 



Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy 

Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The data were weighted to 

account for Interview nonresponse. Standard errors of the estimates account for sample 
weights. 

®AII of the characteristics are based on LIckert scales that measured whether sample members “strongly 
agree,” “agree,” “disagree,” or “strongly disagree” with each statement. Each statement has a maximum 
value of four. For the sake of consistency, each statement was scored In a positive manner. For 
example, a score of 3.0 for the statement “1 feel that 1 am being pushed around In life” means that the 
average client “disagrees” with this statement. In contrast, a score of 3.0 for the statement “1 can do 
anything 1 set my mind to” means that the average client “agrees” with the statement. The statements 
were then summed to calculate the aggregate measures for self-efficacy, self-esteem, and future 
orientation. 

7*7*** Impact estimates are statistically significant at the .05/.025/.01 level, one-talled test. 
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more difficult not only to get a job, but also to stay employed and achieve self-sufficiency. 
Table III. 7 highlights the prevalence of a wide range of challenges that Future Steps might 
have reasonably been able to influence through its case management services. 

D espite the emphasis of Future Steps on helping clients address individual barriers to 
work, similar fractions of program and control group members faced challenges during or 
near the end of the 18-month follow-up period. Logistical challenges, in particular, were 
prevalent for both program and control group members, despite the significant role of 
Future Steps in helping many chents with, and providing supportive service payments for, 
child care and transportation (Table III. 7). At the time of the 18-month followup, more 
than half of program group members (51 percent) reported that they had faced 
transportation problems during the past six months that had made it difficult for them to 
find and keep a job or participate in work activities. Similarly, more than two-fifths 
(42 percent) of program group members reported child care problems during the same time 
frame. The ongoing logistic^ problems that many chents faced suggest that the program 
was not able to address clients' logistical needs in a systematic or lasting way. 



Figure III.7. Number of Major Obstacles and Hardships Among Sample Members 
During the Follow-Up Period^ 

Percentage 




■ Program Group □ Control Group 



Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica 
Policy Research, Inc. 

Note: Estimates were adjusted using multivariate regression methods, and data were weighted to 

account for survey nonresponse. Standard errors of the estimates account for sample weights. 

^Obstacles and hardships include transportation problems; child care problems; fair or poor overall 
health; poor health that inhibits work, training, or school; major depressive disorder; physical abuse by 
spouse or partner; drug or alcohol problems; lack of support or resistance from family or friends; lack of 
health insurance; serious housing problem; and lack of food availability. 

•/**/*** Significantly different from zero at the .05/.025/.01 level, one-tailed test. 
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Table III.7. Differences in the Prevalence of Obstacles and Hardships at the 18-Month Followup 



Characteristic 


Program 

Group 


Control 

Group 


Difference 


Health Problems or Issues® 


Overall health is fair or poor 


29.5 


31.5 


-1.9 


Poor health Inhibits work, training, or school 


25.2 


24.4 


1.8 


Physical disability or Illness 


17.3 


19.4 


-2.1 


Major depressive disorder 


22.7 


19.8 


2.9 


Other Personal Challenges that Hindered Work‘d 
Lack of support or resistance to working from 


family/friends 


15.2 


18.1 


-2.8 


Physical abuse by spouse or partner 


3.0 


3.3 


-0.3 


Drug or alcohol problems 


1.3 


2.7 


-1.5 


Logistical Obstacles That Hindered Work'^ 


Transportation problems 


51.1 


47.7 


3.4 


No access to a vehicle or no driver’s license 


37.5 


33.6 




Child care problems 


42.1 


49.0 


-7.0 


Lack of Health Insurance Coverage 


Uninsured at followup 


39.0 


40.0 


-0.1 


Sometimes uninsured during follow-up period 


56.6 


53.4 


3.2 


Children uninsured at followup 


8.0 


4.8 


3.2 


Children sometimes uninsured during follow-up period 


12.8 


8.8 


4.0 


Housing Issues' 


Lived In public or subsidized housing 


24.4 


27.5 


-3.2 


Could not pay rent or mortgage 


40.1 


39.6 


0.5 


Evicted from home or apartment 


7.6 


8.9 


-1.3 


Could not pay utility bill 


43.4 


41.4 


2.0 


Had utility turned off 


23.7 


20.9 


2.8 


Homeless or lived In a shelter 


9.5 


9.7 


-0.2 


Any serious housing problem 


29.6 


26.2 


3.4 


Food Availability'' 


Food was often or sometimes not available 


46.1 


49.1 


-2.9 


Sample Size 


252 


268 


520 



Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy Research, 

Inc. 



Note: All estimates were adjusted using multivariate regression methods. The data were weighted to 

account for Interview nonresponse. Standard errors of the estimates account for sample weights. 

®The health measures (other than major depressive disorder) represent sample members’ self-reported 
health status at the time of the survey. Major depressive disorder represents the prevalence of major 
depression during the past 12 months, as measured by the World Health Organization’s Composite 
International Diagnostic Interview Short-Form (CIDI-SF) (Kessler et al. 1998). 

“^Other personal challenges and logistical obstacles were those that. In the six months before the survey, 
sample members said made it difficult for them to find and keep a job or participate in work-related activities. 
In particular, for child care problems, sample members were asked If they had any of five different types of 
problems or concerns during the past six months that made It difficult for them to work or prevented them 
from working. In addition, clients’ access to a vehicle or possession of a driver’s license was measured at the 
time of the 18-month survey. 

'^Housing issues were those occurring at any time during the 18-month follow-up period. Clients with any 
serious housing problem had at least one of the following problems during the 18-month follow-up period: 
evicted from home or apartment, had utility turned off, or had been homeless or lived In a shelter. 

''Food availability Issues were those occurring at any time during the 18-month follow-up period. In particular, 
food availability was measured by how frequently “the food that (a sample member’s household) bought did 
not last and (they) did not have money to get more.” 

*r*r** Differences between the program and control groups are statistically significant at the .05/.025/.01 
level, one-talled test. 
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Personal challenges related to health were considerable for a large minority of sample 
members. One-guarter of program group members reported that their poor health inhibited 
their abihty to do certain types or amounts of work, training, or school. In particular, nearly 
one-fifth (17 percent) of program group members reported a physical disability or illness, 
and nearly one-guarter (22 percent) were classified as having had major depressive disorder 
during the past 12 months.^^ The prevalence of these health-related challenges among the 
Future Steps population is generally consistent with national- and state-level estimates of the 
prevalence of similar challenges among the welfare population (Olson and Pavetti 1996; 
Johnson and Meckstroth 1998; Loprest 1999; and Meckstroth et al. 2002).^^ Exacerbating 
the health challenges that many Future Steps clients faced, nearly two-fifths (39 percent) 
were uninsured at followup, and close to three-fifths (57 percent) had been uninsured at 
some point during the follow-up period. 

Both program and control group members faced other hardships that also indicate a 
high level of personal and family need. Table III. 7 shows that more than one-guarter of the 
sample faced a serious housing problem at some point during the foUow-up period. In 
addition, close to half reported that food was often or sometimes not available to them or 
their household during the foUow-up period. 

Differences IN Program Impacts for Key Subgroups 

Although Future Steps had no overall impact on key economic outcomes, we conducted 
subgroup analyses to examine whether the program was effective for certain subgroups of 
the target population. At the evaluation's outset, we knew that the subgroup analyses would 
be somewhat hmited by sample size. We hoped, however, that by examining patterns of 
subgroup effects we would enhance our understanding of the Future Steps program 
experience and how it may have affected client outcomes. G iven the implementation and 
contextual issues described in Chapter II, we focused our analyses on two key subgroups 
defined by chents' (1) period of program enrollment (or year of program participation), and 
(2) level of employabihty. First, we reasoned that a subgroup analysis by the period of 
enrollment might help isolate the effects of the implementation ch^enges that occurred 
during the program's second year. Second, we believed that separate analyses for chents 



11 Major depressive disorder represents the prevalence of major depression during the past 12 months (a 
major episode of depression lasting two or more consecutive weeks), as measured by die World Health 
Organization's Composite International Diagnostic Interview Short-Form (CIDI-SF) (Kessler et al. 1998). 
This battery of questions was included as part of the 18-month follow-up survey. 

12 The prevalence of drug or alcohol problems and physical abuse by a spouse or partner, as shown in 
T able III.7, are quite a bit lower than has been shown in other studies. This is not surprising, given the survey 
questions we us^ to measure these problems. D ue to the length of the survey interview, we were only able to 
ask one question to measure each problem, rather than using a more extensive and well-tested battery 
of questions. 
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who were relatively more or less employable might be useful in understanding how to target 
future program services.^^ 

• Future Steps was better implemented during its Grstyear. Although the findings 
show hints of better program effects for clients served during this year, the 
program-control differences were insignificant and very small. 

During the demonstration's first year (as described in Chapter II), program 
implementation and service delivery were stronger and more consistent with the original 
Future Steps model and vision. The departure of the program director and other staff 
during the demonstration's second year diluted program operations and halted efforts to 
develop customized, employer-focused job trainings. In addition, budgetary pressures that 
IDHS and SCC faced during the second program year exacerbated the effects of the staff 
turnover. 

Because of the stronger implementation during the program's first year, we 
hypothesized that outcomes would be better and impacts would be greater for program 
group members served during the first half of the demonstration.^^ Indeed, a higher fraction 
of the group served during the first year reported receiving employment preparation and 
other services, compared with their program group counterparts served during the second 
year (Appendix C, Table C.IO). Moreover, consistent with our expectation, the significant 
program versus control group differences in service use (described earlier in this chapter) 
were much more pronounced for the group of clients randomly assigned during the first half 
of the demonstration (Table C.IO). 

Nevertheless, despite stronger program implementation and greater service use for the 
first-year program group members, there was no evidence that Future Steps improved 
employment and earnings and reduced welfare dependence for this group. Although the 
pattern of the findings shows hints of better program effects on economic outcomes for 
first-year program group members than for their control group counterparts, the differences 
were statistically insignificant and very small. (Appendix C, Table C.ll summarizes these 
findings using both survey and administrative records data.) 



We also conducted analyses of several other sets of subgroups. We examined impact findings by 
(1) household type (single parents versus all other household types), (2) race/ ethnicity, (3) sex (for females only, 
given the small sample size for males), and (4) relevant groups of counties. The impact findings for key 
outcomes for these groups were neither significant nor informative. 

14 The program was implemented during the two-year period from July 2001 to September 2003, and 
random assignment and program enrollment were conducted during the 18-month period from July 2001 to 
December 2002. We examined impacts separately for program and control group members who were 
randomly assigned during the first hif of the 18-month sampling period (July 2002 to March 2002) and for 
those who were randomly assigned during the second half of the period (April 2002 to D ecember 2002). 
Program group members who enrolled during the first half of the demonstration were served most intensively 
during the first program year (July 2001 to June 2002), while those who enrolled during the second half were 
mostly served during the second program year (the 15-month period from July 2002 to September 2003). 
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• There was no evidence that Future Steps improved employment, earnings, or 

self-sufficiency for either more employable or less employable chents. 

By design, Future Steps services focused most on employment-related services and 
logistical supports that are closely linked to employment. As described in Chapter II, 
employment-related assistance, such as job search and job placement assistance, was the 
most common type of program service chents received. Clients were also more hkely to 
have received assistance accessing logistical supports like child care and transportation than 
they were to have received personal and family services. 

Given the employment focus of the program, we hypothesized that impacts on key 
outcomes might be different for individuals who, at the time of their enrohment, were 
relatively more or less prepared for employment ("more employable" versus "less 
employ^le"). We characterized sample members as less employable if they met at least one 
of the fohowing three criteria: (1) did not have a high school diploma or G ED , (2) had a 
health-limiting condition at the time of random assignment, or (3) had received TANF for 
one or more years during their lifetime.^'’ In contrast, more employable clients did not meet 
any of these three criteria. Nearly three- fifths (58 percent) of the Future Steps sample were 
considered less employable, while two-fifths (42 percent) were more employable. 

The program appeared to provide services to chents based on their need. Trends in 
service use data show, for example, that the significant difference in the fraction of program 
and control group members who received job search and job placement assistance was much 
more pronounced for the more employable chents, who were relatively more prepared to 
move directly into employment (Appendix C, Table C.12). In addition, the less employable 
chents were significantly more likely than their control group counterparts to receive job 
readiness training and to participate in education and training as a way of building skihs and 
preparing for employment (T able C.12). 

Despite such targeting of services, there were no significant improvements in 
employment and earnings for either the more or less employable chents. Appendix C, T able 
C.13 summarizes the findings using both survey and administrative records data. Overah, 
based on a synthesis of the survey and administrative records data, impacts for employment 
and earnings were not significant, and the pattern of the point estimates did not suggest that 
the program might be more appropriate for either the more or less employable chents. 

There was also no evidence of improvement in the level of self-sufficiency for either the 
more or less employable chents. For measures of welfare dependence, the administrative 
records analysis did show that a significantly smaller fraction of the more employable 



Clients with a health-limiting condition were those who responded at baseline that they currently had a 
health problem that limited the kind or amount of work, training, or school work that they could do (problems 
such as a preexisting medical condition, a physical disability, an emotional or mental condition, or drug or 
alcohol use), or that someone else in their household had a disability or health problem that made it difficult 
for them (the sample member) to work, attend training, or go to school. 



Chapter III: Impads on E mployment, Sdf-Suffidency, and W dl-Bang 




70 

program group members, compared with their control group counterparts, relied on TANF 
at some point during the follow-up period (Table C.13). However, this reduced welfare 
dependence did not translate into a significantly smaller fraction receiving TANF or living in 
poverty at the 18-month followup/® In the next chapter, we discuss the implications of the 
findings for designing, implementing, and testing welfare- to -work programs in rural areas. 



The findings also showed that the less employable clients in the program group were more likely than 
their control group counterparts to have received TANF, and to have received it for a longer time, during the 
follow-up period. This suggests that the program, because it had relatively frequent contact with clients, helped 
needy clients obtain the TANF benefits for which they were eligible. 
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C 0 N C L U SI 0 N S AN D RECOMMENDATIONS: 

Improving Program Innovation 
IN Rural Areas 



' I ' he Rural Welfare- to- Work (WtW) Strategies Demonstration Evaluation is a rigorous 
I experiment of promising employment strategies in rural America. The 
i demonstration arose in response to a lack of evidence about how to meet the unique 
needs of low-income workers in rural areas as they progress toward economic independence. 
This report examined the impacts of Future Steps— a promising case management program 
in rural Illinois— on the employment, earnings, welfare dependence, and weU-being of work- 
ready low- income people, including those receiving Temporaiy Assistance for Needy 
Families (TANF) or food stamps. Similar case management models have been tested and 
shown to not be effective in urban areas. Future Steps was an improvement on previous 
models, however, and viewed as potentially effective in rural Ilhnois, for several reasons. 
The model (1) responded to the scarcity of services and jobs by teaming the welfare agency 
with a regional community college, (2) drew on the local connections of program staff, and 
(3) provided very small caseloads. 

D espite the promise of Future Steps, there is no evidence that it improved employment, 
earnings, and self-sufficiency among low- income chents in rural Illinois. Our evaluation did 
show that the core Future Steps case management model was implemented largely as 
intended. Many chents received services at a substantial level of intensity and over an 
extended period, and program group members received significantly more services than 
control group members. Nevertheless, the greater service use did not translate into positive 
impacts on key economic outcomes. Although the findings show hints of better program 
effects on economic outcomes for chents who enrolled and were served during the 
program's better-implemented first year, the magnitude of the effects was insignificant and 
veiy smah. There also was no evidence of program impacts for subgroups of chents defined 
by how employable they were when they enrobed. 

Regardless of the absence of program impacts in this Rural WtW evaluation of Future 
Steps, the evaluation findings stih offer informative, useful lessons to shape future policy and 
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program efforts. In this last chapter, we start by examining potential reasons for the lack of 
program impacts. We then present lessons relevant for implementing programs in a rural 
welfare- to-work context. N ext, we offer recommendations for designing future programs in 
rural areas. Finally, we give suggestions for making future evaluations of rural programs as 
useful as possible. 



Possible Reasons for the Lack of Program Impacts 

Three factors might help explain why Future Steps did not have impacts. The first two 
relate to shortcomings in implementing program services. The third factor, and likely the 
more important, relates to inadequacies in the scope of the program intervention, as 
designed. Each factor might have weakened the ability of Future Steps to effect an overall 
change in chents' employment status and self-sufficiency. 



• Although many chents received substantial services, some received few services 
or insufficient ones. This pattern partly reflects the limited training and support 
given to career speciahsts, who varied in skills and initiative. 

The basic Future Steps case management model was reasonably well implemented. 
Gaps existed in the program's coverage of chents, however, and these gaps might have 
weakened the program's abihty to change chent outcomes. About two-fifths of chents 
received substantial services over an extended period. About one-fifth, however, had little 
contact with career specialists (case managers) and the rest received services at a level that 
may not have gone far enough in connecting them to jobs or helping them overcome 
challenges. Indeed, on the 18-month fohow-up survey, even though service use differences 
between program and control group members were significant, many chents during both the 
first and second years of the program reported that they had not received key types of 
services. Some who did not receive services may not have wanted or needed them. Career 
speciahsts may not have been able to reach others, however, or help them in a meaningful 
way. Service use was particularly low among chents who were never employed during the 
follow-up period, suggesting that the program might have done more to help these chents 
secure gainful employment. 

Future Steps expected a great deal of its career specialists, but it did not prepare or 
support them at a level commensurate with the high expectations. Although the Future 
Steps career speciahsts were dedicated, caring, and hard-working professionals who dehvered 
substantial services to many chents, the incomplete coverage of chents partly reflects the 
limited training, guidelines, and support they received from the program. The career 
speciahsts as a group had hmited education and experience at the outset of the 
demonstration. They also varied in their skills and level of initiative in leveraging their 
community knowledge and connections to help chents. As discussed in the next section, 
increasing career specialist compensation, training, and technical assistance probably would 
have helped improve their skills and the overall delivery of services to chents. Whether such 
improvements would have led to significant gains in clients' employment, earnings, and self- 
sufficiency is uncertain. 
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• Future Steps was not able to capitalize on the employer connections and job- 
training resources that its community college partner offered, thus lessening the 
scope of its job readiness and job placement efforts. 

The infrastructure and expertise of Shawnee Community College (SCC) appeared to 
benefit Future Steps in several ways. As a regional community college, SCC was a primaiy 
provider of education and workforce development services in the region, and local job 
openings were often hsted with the college's placement center. These opportunities were 
shared with Future Steps participants. The college also offered skill assessment and career- 
planning tools useful to the program. Moreover, the college's reputation as a weU-respected 
educational institution helped encourage enrollment and reduce the stigma chents might 
have felt as participants in a welfare-to-work program. To encourage referred chents to 
come to an initial appointment, career speciahsts often emphasized the program's affiliation 
with SCC, downplaying its character as a welfare-related service. Program staff also noted 
that employers were more likely to respond to inguiries from Future Steps when they were 
told of fte program's affiliation with the college. 

Despite these many advantages. Future Steps did not effectively capitahze on the 
employer connections and job-training resources the cohege offered. Future Steps had 
intended, building on the college's vocational-training resources and community 
connections, to work with local employers to implement customized job-training efforts. 
This component of the model was not implemented, however. The program had hoped that 
its employer-focused efforts would help develop job opportunities for chents by preparing 
them for, and connecting them to, good jobs in the local area. The employer-specific 
training, as envisioned, would have included work readiness training, hfe skills building, and 
job-specific vocational instruction, ah of which SCC could help develop and implement. 
This component was not implemented, partly because Future Steps did not invest adeguate 
time and effort into planning for it and building the employer relationships necessaiy to 
support it. 

Two factors might have facihtated the implementation of this program component: 
(1) designating more staff resources for it, and (2) phot- testing it. First, Future Steps only 
invested in a part-time program director. Fuhy capitahzing on the cohege's connections and 
resources would have been easier had Future Steps invested in a full-time director with more 
time to devote to bigger-picture program development issues. Alternatively, the program 
might have designated another sttff person, such as a speciahzed job developer, to lead this 
component. In addition, if this component had been phot-tested, as were the other 
components of the Future Steps model, it would have had a better chance of success. 

• Regardless of program implementation, basic case management is not likely to be 
an adequate intervention for improving employment and self-sufficiency in rural 
areas. Still, it may be an important piece of a stronger intervention. 

The evaluation of Future Steps provided a good test of the effectiveness of basic 
employment-focused case management in a rural setting. The absence of program impacts 
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on employment earnings, and welfare dependence suggests that case managers, no matter 
how sldlled and effective, are hmited in what they can help their chents accomplish. If, as in 
southern Illinois, a local or regional area lacks good jobs, as well as adequate services like 
child care and transportation, then case management services alone appear unlikely to 
overcome such limitations. Indeed, the findings indicate that control group members, 
without the assistance of Future Steps, were just as likely as program group members to 
secure and maintain the mostly low- and semiskilled jobs that were available. Moreover, 
given the small and insignificant magnitude of the program versus control group differences 
in key outcomes, any gains from a better- implemented case management intervention would 
very possibly still have led only to marginal improvements in the economic prospects of the 
work- ready population Future Steps targeted. 

Overall, the evaluation findings suggest that basic case management services are not 
likely to be an adequate intervention to address employment obstacles and to help low- 
income people in distressed rural areas find lasting employment and become self-sufficient. 
This conclusion supports what other studies have already suggested about employment- 
focused case management in urban areas (Rangarajan and Novak 1999). The findings appear 
to suggest a need for additional program elements to help low-income chents in rural areas 
move to employment and self-sufficiency. As examined below, these program elements may 
include strategies to (1) build connections with employers to help identify and develop new 
job opportunities for low-income workers, (2) initiate new economic development 
opportunities in distressed local and regional economies, and (3) expand needed support 
services. This conclusion is most relevant for other rural areas that, hke southern Illinois, 
face substantial economic chahenges, such as high poverty, high unemployment, and a lack 
of good entiy-level jobs that offer decent wages and a chance for benefits and job 
advancement. Such disadvantages may be difficult to overcome without expanded services 
and community development efforts that go beyond case management. 

Case management may still be an important component of a stronger intervention, 
however. Future Steps did offer a useful vehicle for successfuhy delivering substantial 
services to many low-income clients in rural areas. Though not large or significant, the 
magnitude of program effects for clients served during the better-implemented first year was 
somewhat better. This suggests that a reasonably well-implemented case management 
program has some promise for dehvering useful services in rural areas. Case management 
services, though not a program solution on their own, may still be a useful way to provide 
beneficial services to chents, as long as these services are coupled with other efforts that 
address individual and community employment chahenges in a more systemic, 
substantial way. 



Lessons FOR Implementing Future Programs in Rural Areas 

The evaluation findings demonstrate the inherent challenges in helping low-income 
people in rural areas make the transition to employment and self-sufficiency. In rural labor 
markets, good jobs are generahy scarcer than in urban ones, and services and resources may 
be more hmited. In addition, jobs and services can be more difficult to access because of the 
geographic dispersion of people and places. Tight-knit local communities can further 
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impede employment efforts for those with a poor reputation or few ties to a local area. 
These conditions should be considered in the design and implementation of program models 
in rural areas. The experiences from the Future Steps evaluation suggest several lessons on 
program operation and case management service delivery in a rural context. Although the 
potential effect of these lessons or factors is not known, each may be important for 
implementing programs in rural areas successfully. 

• Local staff connections and initiative appear to be important elements of 
successful sendee delivery in rural areas. 

The quality of Future Steps services depended a great deal on the capabihties and 
connections of program staff. The development and early implementation of Future Steps 
was aided by a dynamic, experienced program director. This director communicated an 
ambitious vision to other administrators at the college and ID HS and developed connections 
across the five-county area. The successful day-to-day operation of the program depended 
largely on the abilities of the local Future Steps career specialists, who worked directly with 
clients, developed connections with employers and service providers in their county, and 
helped address cheats' individualized needs. 

The most capable Future Steps career speciahsts appeared to be those who were famhiar 
with their communities and able to identify existing employment opportunities and services 
and make the most of them. Because service providers and employers varied among the 
counties Future Steps served, it was important for individual career specialists to take the 
initiative in developing local connections. Such connections helped their efforts to make 
appropriate referrals for outside services, learn about job openings, actively market clients as 
good candidates for available positions, and, as appropriate, mediate cheats' problems. In 
using their connections to support and advocate for clients, program staff often played an 
important role as a personal reference. Vouching for cheats may have special value in tight- 
knit rural communities, where a poor personal or family reputation can negatively affect a 
person's economic prospects. Such support may make employers more comfortable 
offering cheats a job, can facihtate clients' interactions with other organizations, and can 
build cheats' confidence to continue pursuing employment and other goals. 

• To promote staff recruitment, retention, and a consistently high degree of skill 
and performance, an adequate investment in staff compensation is important 

Program leaders sometimes found it chahenging to recruit and retain career speciahsts 
with the necessary combination of skihs, familiarity with the community, and 
professionalism. As discussed earher, career speciahsts varied in their experience and skih 
level— for example, in their ability to advocate for cheats and leverage av^able community 
resources and opportunities on cheats' behalf. Staff turnover led to short-term breaks or 
slowdowns in service delivery, particularly during the program's second year. 

It was difficult to recruit highly qualified case management staff because most of the 
positions were part-time and did not include a fuh set of employment benefits. Ensuring a 
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high degree of staff skill and performance for new programs will likely reguire a higher- level 
investment in staff, including full-time positions with better compensation. 



• Careful training, oversight, and ongoing support are essential for staff in 

dispersed, rural areas. 

To serve a large geographic area, Future Steps career speciahsts were based in dispersed 
locations separate from program leaders. As a result, they exercised substantial discretion 
and autonomy in their daily work. Career specialists independently managed their schedules, 
prioritized tasks, and acted as representatives of the program in tiieir local community. To 
work effectively with this level of independence, they had to have a high degree of maturity, 
professionahsm, and self-motivation, as well as broad skills. Although the Future Steps staff 
were carefully selected, many had hmited professional employment experience, and most did 
not have a college degree. 

Training, oversight, and ongoing support from program leaders are particularly 
important for guiding the work of staff in dispersed locations, especially those with limited 
professional experience and education. In Future Steps, freguent communication between 
program leaders and staff was a central element of this support. Future Steps administrators 
reviewed career speciahsts' case notes regularly and provided feedback to staff members 
based on this information. Administrators also used freguent email and telephone 
communication to support and monitor staff activities. 

Program leaders could have done more, however, to ensure that Future Steps services 
were of consistently high guality and that staff expertise grew. The initial training for Future 
Steps career specialists was relatively brief and focused largely on administrative issues. The 
training might have been more valuable if it had been expanded to include more mentoring 
from experienced staff and a stronger focus on developing community relationships and 
using local resources. Future Steps career specialists also may have benefited from more 
structured feedback on their work, as administrators did not provide formal performance 
reviews. Such a process might have helped administrators identify staff strengths and 
weaknesses, ensured clear communication between managers and staff, and promoted plans 
to help staff members improve their skihs. 



• incorporating performance incentives into agreements with partner organizations 
may help programs stay focused on goals. 

To ensure successful program implementation, administrators and staff must stay 
focused on the program's goals and objectives. Performance-based contracting can be a 
management tool to encourage program staff to meet predetermined goals and objectives, 
such as those related to job placement, job retention, and the use of supportive service 
funds. IDHS, in its agreement with SCC to operate Future Steps, did not incorporate 
performance-based goals or financial incentives. The familiar relationship and past 
partnership between SCC and IDHS may have precluded the use of performance-based 
management tools in the Future Steps agreement. In addition, SCC might have been waiy 
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of the inherent risks of participating in a performance-based contract related to Future Steps, 
especially since Future Steps was not central to its organizational mission or necessaiy for its 
revenue base. 

For organizations facing resource constraints, as many in rural areas do, performance 
goals and incentives may help program leaders and staff stay focused on achieving goals. 
These tools can be particularly useful when, like Future Steps, programs will not be sustained 
at the end of a funding period. During the demonstration, SCC faced organizational and 
staffing resource constraints. With increasingly constrained resources, SCC administrators 
focused most on program efforts that were most central to the college's educational mission. 
Since Future Steps was not at the heart of SCC's mission, SCC's commitment to the 
program waned, and it did not use all the available Future Steps resources. This reduced 
commitment partly reflected the approaching end of the demonstration. If IDHS had 
incorporated performance incentives or bonuses into its agreement with SCC, however, it 
may have encouraged SCC's Future Steps staff to stay focused on maintaining the intensity 
of services and achieving aU the goals of the program. 



IMPLIC ATIO N S FOR D E SIGN IN G WELFARE -TO -WO RK PROGRAMS IN RURAL AREAS 

One of the fundamental goals of the Rural WtW demonstration is to identify program 
strategies that policymakers should consider in designing future welfare-to-work programs. 
The absence of impacts in this evaluation of Future Steps implies a need for stronger 
interventions that target low-income workers in rural areas. We highlight several program 
strategies below that may help strengthen future welfare-to-work interventions in rural areas. 
Interventions that include one or more of these strategies may be good candidates for 
further evaluation, as their potential effectiveness is uncertain. 



• Building linkages with employers to promote job opportunities may take on 
added importance in rural areas with few good Jobs. Involving job developers in 
program efforts maybe essential. 

By working collaboratively with employers, welfare agencies and their programs may 
help to identify and develop good job opportunities for their clients, and help prepare and 
train clients for jobs. Many welfare-to-work programs, including Future Steps, have found 
that developing employer linkages is challenging (for example, the GAPS and PESD 
postemployment programs— Wood and Paulsell 2000 and Rangarajan and Novak 1999, 
respectively). At a minimum, creating beneficial opportunities for welfare agencies, other 
service providers, and employers reguires thoughtful planning and a substantial investment 
of time and effort. Prospective employers must perceive benefits in working coUaboratively 
with a welfare agency. Like other employers, employers of low-income workers are 
concerned with employees' work attitudes, dependability, and stability; in particular, many 
are skeptical about the work readiness and life skills of welfare recipients (Long and Ouelette 
2004). These concerns, which often are barriers to hiring and retaining welfare recipients 
and other entiy-level workers, could be addressed through closer collaboration, training, and 
job development efforts that involve both service providers and employers. 
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D eveloping relationships with employers and job opportunities for chents may require 
an investment in specialized job development services. Welfare and workforce agencies may 
have more success in collaborating with employers if they hire or contract with professional 
job developers. A job developer with the necessary experience, skills, and creativity to 
develop employer relationships and shape and organize plans for connecting low- and 
semisldUed workers to employers, may have much to contrODute to welfare-to-work efforts. 
As a supplemental strategy to this type of job development work, collaborative efforts 
involving the Workforce Investment Act (WIA) system can provide an entry point to 
working more closely with employers. Demonstration programs in the Employment 
Retention and Advancement Evaluation have found that coordinating services with the 
workforce system often provides access to job leads, employer connections, and WIA- 
funded training (Anderson and Martinson 2003). Euture Steps was not able to engage the 
workforce system in its efforts, and thereby missed an opportunity to capitahze on another 
potential resource. 

• Economic development represents an important strategy for improving the 
employment prospects of low-income workers in distressed rural areas. 

Rural areas with few job opportunities may need to look beyond welfare-to-work 
interventions as a way to help low-income workers find lasting work that boosts their 
prospects for self-sufficiency. Linking TANE and welfare-to-work interventions with 
organizations that provide economic development initiatives may help create new job 
opportunities and connect low- and semiskilled workers to them. As a starting point, when 
welfare and workforce programs collaborate with education and training institutions to focus 
their efforts on the needs of local workers and employers, they may spur economic 
development by making an area more appeahng to new industry. In addition, community 
development efforts in distressed local areas may help to improve the economic prospects of 
low-income workers by alleviating challenges they face related to housing and other services. 

Wage subsidies, tax credits, and low-interest loans to employers are incentives that state 
workforce and welfare agencies and other pohcymakers might consider in rural communities 
and regions that have few good job opportunities. These tools can act as incentives for 
employers to expand their business, create new jobs, and hire low- and semiskilled workers. 
In labor-intensive industries, such tools may have an important influence on business cost 
structures, and prospective employers may be wUhng to consider them as incentives to hire 
the welfare-to-work population. Such incentives might be structured in various ways. Eor 
example, where the number of employees makes it feasible, employer incentives could be 
used to help offset the costs of offering employer-based child care centers or van services. 

• Systematic improvements to the availabihty of logistical services like 
transportation and child care maybe needed in many rural areas. 

Rehable transportation and accessible, good-quality chUd care are logistical supports 
essential to labor market success. Many rural areas lack adequate public transportation and 
accessible, good-quahty child care services (Eriedman 2003; Rucker 1994; Community 
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Transportation Association of America 1996; Casper 1996; Hofferth et al. 1991). Southern 
Illinois is no exception. Clients' service use related to child care and transportation did 
improve as a result of Future Steps. However, the services were not substantial enough to 
fully address the extent of chents' logistical barriers. 

In areas mth a hmited transportation and child care infrastructure, it may not be enough 
to hnk clients with existing services. Rather, systematic improvements that create more or 
better services may be necessary. Expanded transportation and child care— through pubhc 
van service, low-cost car loans, and good-guahty, accessible care during nonstandard work 
hours— may be particularly vital in rural areas with hmited resources. 



Addressing Constraintsof Scale: Issueseor Evaluating Rural Programs 

Conducting evaluations in rural areas is inherently chahenging given constraints of scale. 
Resources, opportunities, and population density are all smaller in rural areas than in urban 
ones. These factors present important issues, not only for program designers and 
administrators, but also for evaluators and funders as they tiy to maximize the usefulness of 
program evaluations in rural areas. The Rural WtW Strategies Demonstration Evaluation 
encountered such constraints of scale in implementing and testing Future Steps. In this 
section, we highhght two key issues— both germane to the Rural WtW experience— that 
evaluators and funders might face when developing future demonstration evaluations in 
rural areas. 

• Evaluators of rural programs can increase the sample size and power of an 
evaluation by expanding a program's enrollment period, target population, 
catchment area, or number of sites. 

Low population densities in rural areas can make it difficult to form treatment and 
control groups that are large enough to detect impacts. During the Rural WtW evaluation's 
site selection phase, it was challenging to identify strong programs where random assignment 
was feasible and where it was possible to form sufficiently large treatment and control 
groups. In the end, the evaluation yielded two tests of promising models— Future Steps and 
Building Nebraska Families. To form sufficiently large evaluation samples, both Ilhnois and 
Nebraska used extended program enrollment periods (18 months for Ilhnois and 28 months 
for Nebraska). In addition, Ihinois broadened its target population to include food stamp 
recipients and low-income volunteers, and Nebraska both broadened the geographic 
catchment area of many of its local sites and added new sites in other rural parts of the state. 
Other promising rural programs could not be considered for the evaluation due to their 
smah scale and the hmited possibihty for expanding their target population, catchment area, 
or number of sites enough to justify an evaluation. 

G iven the importance to an evaluation of generating a large sample size, evaluators of 
promising, new rural programs wih hkely need to think creatively about ways to increase a 
program's scale and an evaluation's sample size. This might be done by extending the 
program enrohment period, expanding the target population, increasing the catchment area. 
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or adding program sites. In particular, enlarging the program by implementing it in 
additional sites (for example, in neighboring counties or in other parts of the same state) 
might be the most effective way to substantially increase an evaluation's sample size. If 
additional sites are added, however, more central guidance and oversight will hkely be 
needed to ensure that sites consistently implement the same program model. If an evaluation 
has the resources to expand a program in this way, it would likely help to broaden the 
number of promising, small-scale programs that could be considered for future evaluations. 
Increasing the sample size would also improve the breadth and precision of the subgroup 
analyses and the conclusiveness of the findings. The program design and implementation 
lessons from this and forthcoming reports from the Rural WtW evaluation will help identify 
promising rural program models that may be well suited for future evaluations. 

• To strengthen program models and their implementation, future evaluations in 
rural areas might beneSt horn more intensive technical assistance before and 
during a demonstration. 

The provision of technical assistance may take on added importance in rural areas, given 
limits in local resources. Demonstration programs in both urban and rural areas can benefit 
from technical assistance on program development and implementation as a way to build 
strong programs and stay focused on program goals. Rural programs, however, may have 
relatively more to gain. Rural areas may generally have fewer resources to invest in the 
development of new and innovative program models. Moreover, when rural programs 
encounter operational challenges, it can be relatively difficult to recover and adapt guickly 
because of fewer local resource choices and options. For example, when staff resign or turn 
over, it is difficult to identify and hire new staff guickly because the pool of gualified 
replacement staff may be relatively limited. 

More active technical assistance before and throughout the demonstration might have 
helped Future Steps develop its employer-focused job training component and, more 
generally, strengthen the implementation of its case management program model. The Rural 
WtW evaluation included a technical assistance component. By design, however, that 
component related to implementing evaluation procedures and providing oversight of 
program operations mostly during the early phases of operation. In hindsight, it would have 
been more costly, but probably beneficial, if the evaluation design had (1) focused greater 
attention on strengthening the development of the program before the start of random 
assignment, and (2) incorporated more freguent and intensive on-site monitoring and 
assistance throughout the period of program operations. D oing so may have promoted the 
development and operation of a stronger program model. 
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Appendix A 



Survey Data Collection and 
Weighting Methods 



' I 'his appendix describes (1) the methodology used as part of the Rural Welfare-to- 
I Work Evaluation's 18-month telephone survey of the lUinois Future Steps sample, 
i and (2) the procedures used to weight these survey data. 

Survey Methodology 

In this first section, we describe the methods used to design and conduct the 18-month 
follow-up survey. In particular, we discuss (1) sample disposition and completion rates, (2) 
sample selection and enrollment processes, (3) the survey instrument design process and the 
pretest, (4) interviewer training and quahty assurance, and (5) data collection and locating 
procedures. 

Sample Disposition and Completion Rates 

For the 18-month foUow-up survey of Future Steps sample members, we attempted to 
complete an interview with all 630 sample members 18 months after they were randomly 
assigned into the study. The survey was primarily conducted by telephone, assisted by field 
locating: nearly three-quarters of the surveys were conducted by interviewers in MPR's 
telephone center, and more than one-quarter originated from field locators equipped with a 
cellular telephone that the sample member could use to call in to the MPR telephone center. 

The overall survey completion rate was 83 percent (T able A. 1). We completed a total of 
520 surveys (out of 630 sample members)— 379 originating from MPR's telephone center 
and 141 originating from cellular telephones used by field locators. The response rate for 
control group members was somewhat higher than that for treatment group members (85 
versus 81 percent). Among the 110 sample members who did not complete interviews, 79 
were not beatable; 17 refused to do the interview; 7 were located, but we were unable to 
contact them after many attempts; 6 were incarcerated; and 1 was deceased. There was very 
httle difference between treatment and control group members in the reasons why surveys 
were not completed. We discuss patterns of survey nonresponse in greater detail in the 
section below on data- weighting procedures. 
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Table A.1. Final Disposition of Cases for the Rural Welfare-to-Work 18-Month Follow-Up 
Survey of Future Steps Sample Members 



Final Status of Cases 


Treatment Group 
(Number 
(Percentage) 


Control Group 
(Number 
(Percentage) 


Total 

(Number 

(Percentage) 


Total Completes 


252 (80.5) 


268 (84.5) 


520 (82.5) 


Complete (Telephone) 


186 (59.4) 


193 (60.9) 


379 (60.2) 


Complete (Field) 


66 (21.1) 


75 (23.7) 


141 (22.3) 


Refusal 


11 (3.5) 


6(1.9) 


17(2.7) 


Incarcerated 


5(1.6) 


1 (0.3) 


6(1.0) 


Deceased 


1 (0.3) 


0 (0.0) 


1 (0.2) 


Unable to Locate 


40 (12.8) 


39 (12.3) 


79 (12.5) 


Located, Cannot Contact 


4(1.3) 


3 (0.9) 


7(1.1) 


Sample Size 


313 


317 


630 



Note: The survey was conducted by Mathematica Policy Research, Inc. 



Special challenges are associated with interviewing sample members in rural areas. 
These include issues with telephone coverage, transportation, and geographic distances that 
make completing interviews difficult. Planning for the survey incorporated procedures to 
account for these challenges and maintain an acceptable completion rate. During baseline 
interviews, extensive contact information was collected from sample members. Before the 
start of interviewing, prehminaiy database searches were conducted for all sample cases to 
identify those that reguired more locating work. The amount of time cases were worked in 
our telephone center was hmited, to allow more time for field locators to work the cases. 
Field locators were recruited locally, so that they would be familiar with the local geography 
and not be intimidating to the sample members. Ilhnois Department of Human Services 
(IDHS) staff also provided us with monthly address updates on active sample members. In 
addition, a $20 incentive was offered to all sample members for completing the interview. 
We discuss these various steps in more detail later. 



Sample Selection and Enrollment 

The sample consisted of all individuals referred to the Future Steps program during the 
18-month enrollment period who were ehgible to receive services. People were randomly 
assigned to either the treatment group, whose members were eligible to receive the full range 
of program services, or to the control group, whose members received only those services 
av^able outside of Future Steps. 

0 ur goal was to recruit at least 600 sample members and achieve a survey response rate 
of 85 percent (510 completes). We randomly assigned 630 sample members in Ilhnois and 
completed interviews with 83 percent of them (520 completes). 
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Our enrollment process consisted of the following steps performed by IDHS staff in 
each of the five county offices where Future Steps was implemented: (1) completion of 
enrollment paperwork (including informed consent basehne information form, and contact 
information); (2) submission of sample members' information forms for random assignment 
processing through the Interactive Voice Response System (IVRS), which was managed and 
overseen by MPR; (3) notification of sample members about their random assignment 
outcome; and (4) entry of treatment group member information into the Future Steps 
Information System (FSIS). AU the hard-copy forms were shipped to MPR for data entry 
and storage. 

IDHS local office staff received extensive training from MPR on how to administer the 
baseline enrollment forms. These forms included the informed consent form, baseline 
information form, and contact information form. After all these forms were administered to 
sample members and checked for guahty, the IDHS staff telephoned the IVRS and entered 
key pieces of information about the applicant. After the IVRS determined that the apphcant 
was not a duphcate and was eligible for the program, that applicant was randomly assigned 
to either the treatment or control group. The system instantly reported the outcomes to the 
IDHS staff person, who recorded them on the forms. This process minimized the amount 
of extra work IDHS site staff had to complete. It also allowed them to know the outcome of 
the random assignment process almost immediately. 

All forms went through a rigorous guality control process after they were returned to 
MPR. Missing or incorrect data were retrieved from the sites or, in many cases, the sample 
members themselves. AU forms were data entered with 100 percent verification. 



Survey Instniment Design and Pretest 

The survey instrument was designed to be administered by computer- assisted telephone 
interviewing (CATI), with follow-up work by field locators using cellular telephones. The 
survey was designed to take 45 minutes for the respondent to complete. A paper- and-pencU 
version of the instrument was also developed for use in situations where telephone 
administration was impractical, such as prisons or areas not covered by ceUular telephones. 

In designing the survey, we drew heavily from guestionnaires and instruments used in 
previous studies. We also consulted two outside experts: (1) Bruce Weber from the 
Department of Agricultural and Resource Economics at Oregon State University, and (2) 
Greg Duncan from the Joint Center for Poverty Research at Northwestern University. The 
instrument uses guestions from the National Evaluation of the Welfare- to- Work Grants 
Program, the National Job Corps Study, the National Longitudinal Survey of Youth 1979, 
the National Survey of America's Famihes, the Current Population Survey, the Iowa Core 
Survey of Current and Former TANF Recipients, the Iowa Child Impact Study, the 
Postemployment Services Demonstration, the 1998 Survey of Former AFDC Recipients in 
Milwaukee, the Voices of Rural America National Survey, the Nebraska Welfare Evaluation 
Client Survey, the Survey of New Parents, and the World Health Organization's Composite 
International Diagnostic Interview Short Form (CIDI-SF). In addition, many new items 
were created specifically for this instrument. The survey was drafted between Februaiy and 
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April 2002 and submitted to ACF for review. It was revised based on feedback from ACF 
and our consultants. 

We conducted a survey pretest to identify ways to improve (1) the flow and sequencing 
of questions, (2) administration procedures, (3) length of the survey, (4) wording of the 
questions, and (5) instructions for the interviewers. During August and September 2002, we 
pretested several versions of the survey. We completed six pretest surveys in total. The 
participants in the six pretests included people drawn from the predemonstration Rural 
WtW programs in Illinois, Nebraska, and Tennessee. The interviews were drawn from aU 
three sites to simulate the hkely disposition of the full Rural WtW sample.^ 

We trained three experienced interviewers famihar with the evaluation to complete the 
pretest interviews. One interviewer worked primarily evenings and weekends, while the 
other two worked mornings, afternoons, and some evenings. We felt that the schedules of 
these interviewers would give us the best coverage for this population. The six completed 
interviews averaged 67 minutes. We modified the instrument in an iterative fashion, based 
on information obtained through survey monitoring by MPR researchers and debriefings 
with interviewers. Because the interview took longer than expected, we cut many questions 
from the instrument. We also made adjustments to several items based on respondents' 
abihty to understand the questions and answer them. 

After completing the pretest, we submitted the follow-up survey instrument and 
supporting materials to the Office of Management and Budget (0MB) for approval. Based 
on their comments, we made additional revisions to the instrument before the start of data 
collection. 

Although a CATI apphcation was used for the actual data collection, the survey pretest 
was conducted using a paper- and-pencil version of the instrument. Because of the extensive 
programming that would be required to make the many rounds of CATI revisions during the 
pretest, it was not practical to program and test a CATI version of the instrument. The 
CATI apphcation was developed after we made final revisions to the instrument. The length 
of the final instrument, administered via CATI, was an average of 51 minutes. 

Interviewer Training and Quality Assiuance 

Before the start of data collection, we held trainings at our telephone center for all MPR 
project staff. In attendance were the project director, the survey director, and a survey 
assistant. Ah telephone interviewers and locators were required to attend a 12-hour training 
designed to give them a thorough understanding of the project goals and the skills necessary 
to produce good-quahty data. Ah survey supervisors and monitors also received training so 
they could monitor the quahty of the data collection. 



1 The same survey is being used to collect follow-up information from the evaluation's Building N ebraska 
Families sample. 
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Training included a broad range of topics. Trainees were given background information 
on the study, including information about its research goals. The survey instrument was 
reviewed, item by item, with detailed explanations about the meaning and correct 
administration of the guestions. Trainees also received instruction on sample management, 
strategies for contacting sample members and explaining the study, and guidelines for 
appropriate guestion probing. Before completing the training, each trainee was expected to 
complete two practice interviews, monitored by project staff. 

As part of our regular guahty assurance procedures, we conducted ongoing survey 
monitoring for all active interviewers. Each interviewer was monitored on approximately 10 
percent of his or her calls, including introductions and survey refusal conversion attempts. 
Our professional survey monitoring staff, as well as Rural WtW project staff, monitored 
interviewers over the entire duration of the study. 

We hired field locators to work on cases that we were unable to locate from our 
telephone center. We hired local residents and trained them in intensive locating technigues. 
Only minimal training on the instrument was reguired, since the locators' primaiy 
responsibility was to find sample members and then encourage them to call the telephone 
center. Local staff were famibar with the geography and were better able to plan trips to 
maximize their coverage. They were also famihar with local customs and were able to build 
rapport with sample members more guickly. In addition, they were able to connect with 
sample members' friends and relatives to obtain their help locating the sample members. 

For interviews initiated through a field locator, we routinely verified 10 percent of the 
locators' completed cases. Completed cases were randomly selected for either telephone or 
mail vahdation, in which the respondent completed a short guestionnaire, confirming that he 
or she had completed the interview and was a member of the research sample. 

Data Collection and Locating Procedures 

The survey data were collected during the 18-month period from March 2003 to August 
2004. Before the start of data collection, we reviewed the sample cases that had been 
randomly assigned to date and identified sample members with changed or incomplete 
contact information. We rehed on several national databases, comparing our sample to 
existing contact information and updating our records with any new information. This 
useful step was repeated periodically throughout sample enrollment, as new cases were 
added to the sample. 

Given the time- sensitive nature of the survey, cases were released to the telephone 
center exactly 18 months from their date of random assignment. Because sample enrollment 
was spread over many months, we used hard-copy contact sheets to manage the sample 
flow. We generally worked cases in the telephone center for approximately six to eight 
weeks. For those not completed at the end of that period, we began field locating and 
followup. 

We mailed an advance letter one week before the target date on which we would initially 
can a sample member. The letter introduced the study, explained MPR's role in it, and 
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invited the sample member to call us on our toll-free line and participate in the survey at 
their earliest possible convenience. It offered sample members a $20 incentive for 
completing the interview and explained that participation was voluntary and that the 
identities and responses of all participants would be kept confidential. Through the advance 
letters, we also identified cases with incorrect contact information. Some of the letters were 
returned to us because of out-of-date address information, and others were returned with 
forwarding address information. We remailed the letters with new information to the new 
addresses and updated our records with the new information. Those letters without new 
information reguired additional locating. 

The next interviewing step involved calling each sample member on his or her target 
interview date to attempt to complete an interview. If the interview could not be completed, 
appointments for future interviews were made when possible. Alternatively, we scheduled 
routine foUowup of these cases on varying days and times. If the initial contact attempt 
identified sample members with incorrect telephone numbers or outdated contact 
information, these cases were immediately tagged for additional locating. 

We used several technigues to locate sample members whose contact information was 
out-of-date. We contacted family members and friends for updated contact information. 
Failing that, sample members' identifying information was run through several national 
databases owned by LexisNexus. Using names, social security numbers, dates of birth, and 
last known addresses and telephone numbers, new contact information was generated for 
interviewers. In addition, to tiy to identify sample members who might have become 
incarcerated since enrolling in the program, locators searched Internet databases with federal 
and state corrections information. Moreover, IDHS staff provided us with monthly address 
updates for the outstanding sample members on our hst. This assistance from IDHS was 
invaluable to the success of the study. 

We mailed letters and postcards to sample members with whom we had not completed 
interviews. Every few months, we changed the format and content of the letters and 
postcards, as well as the size and appearance of the envelope and the method of mailing 
(regular first class mail versus piioiily mail). We did this to spark sample members' interest 
in opening the letter and reading it. 

A small number of sample members initially refused to participate in the survey. After 
their initial refusal, we waited a week, then mailed them a personalized, specially crafted 
letter designed to change their mind about participating. The letter reiterated the importance 
of the study and their participation in it. They were invited to call our toU-free number to 
complete the interview and reminded of the $20 incentive. We waited until we were 
confident they had received the letter, and then a specially trained "refusal conversion 
interviewer" called to attempt to gain the sample member's cooperation. If this attempt 
resulted in a second refusal, the case was sent to the field, to be attempted in person. In- 
person refusal conversion attempts are often more successful, since there is a personal 
connection, and the respondent feels important because of the extra effort made. 
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Data-Weighting Procedures 

In this section, we describe the evaluation's data- weighting procedures. We begin with 
an analysis of patterns of nonresponse in the 18-month survey data and follow with a 
discussion of the nonresponse adjustments made in the computation of the survey weights. 



N onresponse Patterns 

The Future Steps population had 630 eligible cases. AH the eligible cases were used in 
the study, and 520 of them responded to the survey. We compared the characteristics of the 
survey respondents with those of the nonrespondents to examine differences between them. 
0 ur analysis showed that there are not significant statistical differences in the distribution of 
the respondents and the nonrespondents along key baseline characteristics (T able A. 2). 

The response rate for the survey was 83 percent (T able A. 3). There is a small difference 
in response rates (4 percentage points) between the treatment (81 percent) and the control 
(85 percent) groups. The largest difference of 12 percentage points is between the males (73 
percent) and females (85 percent), and the second-largest difference of over ten percentage 
points is between the cases who are not currently working (81 percent) and the cases who 
are currently working (91 percent). 

If the participants in the study are divided into smaller groups (treatment versus control 
crossed with male/ female, or crossed with male/ female and currently working/ not currently 
working), we find still larger differences among the response rates. For example, the 
response rate for males in the treatment group is 71 percent (the response rate for the males 
not working is veiy close to all the males, because the number of males working is very small 
and it can not be generalized). The response rate for working females in the treatment group 
is 98 percent. 



Computation of the Weights 

The weights were computed using two components, both of which accounted for 
survey nonresponse. We developed two separate weighting adjustments: (1) a weighting cell 
adjustment for nonresponse, and (2) a poststratification adjustment to mimic the 
demographic population characteristics under study. Because we have a census, not a 
sample, of ehgible program participants, the base weight for all cases is one. These two 
adjustments comprise the find weight. 

For the first adjustment, we formed weighting cells within the treatment and control 
groups using the characteristics that best describe the completion pattern— gender and 
worldng status at the time of random assignment— with a minimum of 20 completed cases 
for each cell. The cells with more than 20 completed cases are males in the treatment group, 
females not working in the treatment group, females working in the treatment group, males 
in the control group, females not working in the control group, and females working in the 
control group. Each cell had as a nonresponse adjustment the ratio of the participants in the 
study to the number of responding participants in the ceU. For example, there are 180 
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Table A.2. Comparison of Survey Respondents with Nonrespondents 





Respondents 


Nonrespondents 


Characteristics at Baseline 


Counts 


Percentage 


Counts 


Percentage 


Treatment or Control 


Treatment 


252 


48 


61 


55 


Control 


268 


52 


49 


45 


Gender 


Male 


108 


21 


39 


35 


Female 


412 


79 


71 


65 


Race 


Black 


273 


53 


50 


45 


Non black 


247 


48 


60 


55 


Ethnicity 


Fllspanic 


17 


3 


4 


4 


Non-Hlspanic 


486 


93 


104 


95 


Unknown 


17 


3 


2 


2 


Age at Enrollment 


Younger than 20 


39 


8 


5 


5 


20 to 29 


240 


46 


55 


50 


30 to 39 


153 


29 


35 


32 


40 or older 


88 


17 


15 


14 


Education 


No GED or high school diploma 


132 


25 


30 


27 


GED or high school diploma 


230 


44 


46 


42 


More than high school diploma or GED 


158 


30 


34 


31 


Flousehold Composition 


Single adult 


379 


73 


79 


72 


Multiple adults 


132 


25 


29 


26 


Unknown 


9 


2 


2 


2 


Age of Children 


Less than 3 years old 


150 


29 


32 


29 


3 to 5 


81 


16 


17 


15 


6 to 17 


137 


26 


27 


25 


More than 18 


147 


28 


34 


31 


Unknown 


5 


1 


0 


0 


Currently Working for Pay 


Yes 


105 


20 


10 


9 


No 


415 


80 


100 


91 


Currently Receiving TANF 


Yes 


82 


16 


13 


12 


No 


431 


83 


96 


87 


Unknown 


7 


1 


1 


1 



Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica 
Policy Research, Inc. 

Note; We conducted chl-squared tests for all of the characteristics to test for differences between 
respondents and nonrespondents. 

7*7*** Significantly different from zero at the .10/.05/.01 level, two-talled test. There were no significant 
differences between the two groups. 
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Table A.3. Response Rates, by Key Baseline Characteristics 





Population 


Respondents 


Response Rate 


All 


630 


520 


82.5 


Male^ 


147 


108 


73.5 


Female 


483 


412 


85.3 


Not currently working for pay'’ 


515 


415 


80.6 


Currently working for pay 


113 


103 


91.2 


Treatment 


All 


313 


252 


80.5 


Male 


76 


54 


71.1 


Female 


237 


198 


83.5 


Female not working 


180 


142 


78.9 


Female working 


56 


55 


98.2 


Control 


All 


317 


268 


84.5 


Male 


71 


54 


76.1 


Female 


246 


214 


87.0 


Female not working 


201 


175 


87.1 


Female working 


44 


38 


86.4 



Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by 
Mathematica Policy Research, Inc. 

®The number of males In the treatment and control group Is very small (76 for the treatment and 71 
for the control group). In addition, there were only six males In the treatment group who were 
working at baseline, and seven In the control group. We did not specify the population counts, the 
number of respondents, or the response rates for such small subgroups. 

‘’Data were missing for sample members’ baseline employment status for two cases. 



nonworking females in the treatment group, and 142 of them completed the survey. The 
nonresponse adjustment for the 142 females who responded to the survey is 180/ 142 = 
1.26761. The nonresponse adjustment was apphed to all completed cases to compensate for 
the noncompleted cases. 

The second adjustment was a poststratification of the completed cases by treatment or 
control group and gender and race (considering race as black or nonblack). We used the 
previously described nonresponse adjustments as the weight for the respondents in each ceU. 
Each cell had as its poststratification adjustment the ratio of the population for that gender, 
race, and treatment group to the weighted number of responding participants for that 
gender, race and treatment group. For example, there are 117 black females in the treatment 
group, 100 of them responded, and the sum of their adjusted weight is 118.269. Then, the 
poststratification adjustment for the black females in the treatment group is 117/ 118.269 = 
0.98927. The final weight for each respondent is the poststratified nonresponse adjustment. 
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Overall, the nonresponse adjustments for the treatment and control groups created a 
small design effect close to one due to unequal weights. The effective total survey sample 
size for the treatment and control groups are 248 and 267, respectively, compared with an 
actual sample size of 252 and 268, respectively. 
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Additional Cost Study Analyses: 
Costs by Program Component 



A s part of the Future Steps program cost study, we allocated total costs to six 
components that represent the key services and activities of the program. We 
selected the six program components (see text box below) not only for their 
relevance to Future Steps, but also so we could compare Future Steps component costs with 
those from other welfare-to-work initiatives. In this appendix, we describe the methods we 
used to estimate the costs of the program components, as well as key findings related to 
them. 

Methods 

We allocated total program costs to each of six key components, primarily by examining 
how program administrators and staff spent their time and used program resources. 
Accounting records were not available by program component. Therefore, on the site visits, 
we talked extensively with administrators and staff, asking them to reflect on the fraction of 
time they spent on each program activity, taking into account variations across the weeks 
and months during the cost period. We identified an appropriate fraction of time spent per 
component for each staff member, then used these fractions to allocate total labor costs 
across components. 

For nonlabor costs, to the extent possible, we allocated costs to components based on 
their direct association to a given component. For example, the dollar value of the 
supportive service payments made to clients was allocated to the supportive service 
component. In other cases, when nonlabor costs were not directly associated with a 
particular component, we allocated costs to components in the same proportion as the 
component's share of total labor costs. Allocating costs to components was inherently 
inexact, particularly given that some of the components included closely related activities or 
functions. However, we beheve that the results provide useful and reasonably accurate cost 
estimates for the most important elements of the Future Steps program. 
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Key Program Components 

• Outreach and Recruitment All activities specifically aimed at publicizing 
program services and generating referrals or enrollment, including the time 
Future Steps staff spent coordinating with ID HS staff to enroll chents. 

• Case Management Assistance. G eneral assistance, including orienting 
participants to the program, conducting assessments, coordinating service 
referrds and receipt of services, and monitoring progress. 

• Job Readiness/ Life Skills Training. Specific assistance to enhance the 
employabihty and job search preparedness of participants— for example, 
coaching and mentoring on workplace readiness issues and life sMs, 
conducting mock interviews, and developing resumes. 

• Job Search/ Job Placement/ Postemployment Support. AU activities to 
identify job openings and help participants secure employment— for 
example, helping participants examine job hstings and obtain jobs, 
conducting on-site contacts with prospective employers, and providing 
postemployment support to resolve workplace issues, develop skills, and 
obtain a better job. 

• Supportive Service Payments. The actual value of the payments, along 
with time spent coordinating with clients to identify supportive service 
items, seeking approval from program administrators for payments, 
obtaining items, and completing reguired paperwork. 

• Program Management Program oversight, record keeping, updating the 
FSIS management information system, and general administrative duties. 



Findings ON Program Component Costs 

Individuahzed case management, job readiness training, and job search, placement, and 
postemployment support represent the foundation of Future Steps services. More than half 
of all program costs (53 percent) were associated with this complementaiy trio of 
employment-related services (Figure B.l). These three components were intended to engage 
participants in program activities, help them prepare for and secure a job, and support them 
after they began work. Viewed alone, services directly related to helping chents get and keep 
a job (job search/ placement/ postemployment support) represent the most costly program 
service— nearly one-guarter (23 percent) of total program costs. 

The scope of the employment-related services that Future Steps provided was 
somewhat lower than that of other recent welfare-to-work initiatives. Findings from the 
cost study of the Evaluation of the Welfare to Work (WtW) Grants Program offers a useful 
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Figure B.1. Allocation of Costs Across Future Steps’ Program Components 
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Source: Cost estimates developed by Mathematioa Policy Research, Inc., as part of the Rural Welfare-to-Work Evaluation. 



point of comparison (Perez-Johnson et al. 2002). Nine of the programs in the WtW cost 
study implemented an "enhanced direct employment (EDE)" model comparable to Euture 
Steps. Like Euture Steps, these EDE programs operated outside of the welfare agency, 
focused on moving participants into unsubsidized employment quickly, and enhanced 
traditional work first interventions with a range of services. Among the nine EDE 
programs, an average of two-thirds (67 percent) of total costs were associated with case 
management, job readiness, and job placement, compared with 53 percent for Euture Steps 
(Perez-Johnson et al. 2002). Compared with these other programs, Euture Steps placed less 
emphasis on job readiness and life skills training and more emphasis on supportive service 
payments. 

Supportive service payments were a central aspect of the Euture Steps program. Euture 
Steps often used supportive service payments to deviate clients' widespread transportation 
problems. Supportive service payments, and the labor costs associated with authorizing and 
obtaining them, represent more than one-quarter (26 percent) of Euture Steps costs, making 
it the single most costly program component. 
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Smaller, but essential, program costs related to program management and the outreach 
to, and recruitment of, new chents. Management costs in Future Steps equaled 15 percent of 
total program costs. These costs reflect the involvement of both SCC and IDHS staff in 
implementing and overseeing the program and the time many staff spent in keeping the FSIS 
information system up-to-date. Only a small fraction (six percent) of program costs 
supported outreach and recruitment. These low outreach and recruitment costs reflect the 
ease with which IDHS and SCC caseworkers were able to recruit new clients into the 
program. This cost component also includes the program's use of a small pool of 
discretionaiy funds to provide supportive services to the general SCC student population as 
a way to market the program and promote recruitment. 
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Figure C.1. Participation in Education and Training During the 18-Month Foiiow-Up Period 
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■ Program Group □ Control Group 



Source: Based on the Rural Welfare-to-Work 1 8-Month Follow-Up Survey, conducted by Mathematica 
Policy Research, Inc. 

Note: Estimates were adjusted using multivariate regression methods, and data were weighted to 

account for survey nonresponse. Standard errors of the estimates account for sample weights. 

*/**/**• Significantly different from zero at the .05/.025/.01 level, one-tailed test. 



Figure C.2. Logisticai Assistance During the 18-Month Foiiow-Up Period 

Percentage 
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to 
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Help Finding or Help Finding or Help Finding Job- 

Making Child Care Making Transportation Related Clothing, 

Arrangements Arrangements Tools, or Supplies 



■ Program Group □ Control Group 



Source: Based on the Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica 
Policy Research, Inc. 

Note: Estimates were adjusted using multivariate regression methods, and data were weighted to 

account for survey nonresponse. Standard errors of the estimates account for sample weights. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-tailed test. 
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Table C.1. Transportation and Other Logistical Support Services Received During the 



18-Month Follow-Up Period 



Outcome (Percentages) 


Program 

Group 


Control 

Group 


Difference 


Help Finding or Making Transportation Arrangements 


26.5 


13.3 


13.3*** 


Help Paying for Transportation from an Agency 


30.4 


12.7 


17.7*** 


Gas vouchers 


22.4 


3.7 


18.7*** 


Money to register car, get Insurance, obtain license 


8.8 


2.1 


6.7*** 


Money for car repair or maintenance 


9.8 


3.9 


5.8*** 


Voucher/passes for bus, taxi, van, or train 


7.8 


2.3 


5.5*** 


Money to purchase a car 


5.1 


0.4 


4.8*** 


Help with Transportation from an Employer 


5.2 


3.4 


1.8 


Help with Transportation from Other Sources® 


Received gifts from famlly/friends to help purchase a car 


9.1 


9.6 


-0.5 


Received gifts from famlly/friends to help pay for 


expenses related to car repair/maintenance 


11.1 


11.2 


-0.2 


Obtained a loan to help pay for car 


8.5 


11.3 


-2.7 


Sampie Size 


252 


268 





Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy 

Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The data were 

weighted to account for Interview nonresponse. Standard errors of the estimates account 
for sample weights. 

^Glfts from “family or friends” represent only family or friends who were not living In the sample 
member’s household. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-talled test. 
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Table C.2. Monthly Employment Rates Based on Survey Data (Percentages) 



Outcome 


Program 

Group 


Control 

Group 


Estimated 

Impact 


Whether Employed, by Month After Random Assignment 
1 


31.2 


31.3 


-0.1 


2 


40.4 


35.3 


5.1 


3 


43.9 


37.9 


6.0 


4 


47.5 


40.1 


7.4* 


5 


46.1 


43.2 


2.9 


6 


46.6 


44.6 


2.0 


7 


49.1 


47.0 


2.1 


8 


51.4 


48.9 


2.5 


9 


47.1 


50.1 


-3.0 


10 


49.4 


52.4 


-3.0 


11 


50.2 


52.2 


-2.0 


12 


51.4 


52.8 


-1.3 


13 


50.9 


52.2 


-1.3 


14 


53.6 


54.8 


-1.2 


15 


53.4 


54.7 


-1.3 


16 


53.5 


54.9 


-1.4 


17 


54.7 


56.5 


-1.8 


18 


54.2 


54.9 


-0.6 


Ever Employed During 18-Month Follow-Up Period 


74.5 


75.1 


-0.6 


Sample Size 


252 


268 





Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy 
Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The data are 

weighted to account for Interview nonresponse. Standard errors of the estimates 
account for sample weights. 

7*7*** Impact estimates are statistically significant at the .05/.025/.01 level, one-talled test. 
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Table C.3. Employment Rates and Earnings Based on Administrative Data 



Outcome 


Program 

Group 


Controi 

Group 


Estimated 

Impact 


Employment (Percentages) 

Empioyed, by quarter after random assignment® 


1 


47.8 


38.5 


9.3*** 


2 


46.4 


39.6 


6.8* 


3 


46.0 


39.4 


6.6* 


4 


41.6 


40.3 


1.3 


5 


39.6 


37.0 


2.6 


6 


35.6 


36.5 


-0.9 


Ever empioyed, year 1 after random assignment 


66.6 


64.2 


2.4 


Earnings (Dollars) 

Earnings, by quarter after random assignment 


1 


594 


490 


104 


2 


767 


798 


-30 


3 


831 


741 


91 


4 


800 


877 


-77 


5 


837 


830 


7 


6 


879 


887 


-8 


Totai earnings, year 1 after random assignment 


(average) 


2,992 


2,905 


87 


Sample Size 


313 


317 





Source: Administrative records data from the state of liiinois, compiied by Mathematica Poiicy 
Research, Inc. as part of the Rurai Weifare-to-Work Evaiuation. 

Note: Aii estimates were adjusted using muitivariate regression methods. 

^We present quarteriy empioyment and earnings data for six quarters, or 18 months. However, 
the data for the iast two quarters (quarters 5 and 6) are incompiete and shouid be treated with 
caution. The empioyment rates for these two quarters are reiativeiy iow due to the incompiete 
data. However, we have inciuded these preiiminary data since we assume that the data are 
equaiiy incompiete for the program and controi groups. The preiiminary data for these iast two 
quarters suggest that the eariy program versus controi differences in the empioyment rate do not 
persist during the iast two quarters of the 18-month foiiow-up period. 

7*7*** Impact estimates are statisticaiiy significant at the .05/.025/.01 ievei, one-taiied test. 
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Table C.4. Monthly and Total Earnings Based on Survey Data 



Outcome 


Program Group 


Control Group 


Estimated 

Impact 


Average Earnings per Month, by Month 
After Random Assignment (Dollars) 


1 


281 


293 


-12 


2 


377 


387 


-10 


3 


434 


430 


4 


4 


459 


459 


0 


5 


470 


489 


-19 


6 


499 


492 


7 


7 


543 


521 


22 


8 


540 


550 


-10 


9 


506 


567 


-61 


10 


525 


591 


-66 


11 


543 


589 


-46 


12 


575 


600 


-25 


13 


555 


595 


-40 


14 


544 


590 


-46 


15 


552 


592 


-40 


16 


550 


593 


-43 


17 


563 


589 


-26 


18 


555 


593 


-38 


Earnings per Month During the 
Entire 18-Month Period (Percentages) 


0 


24.3 


21.5 


2.8 


$1 to $249 


14.8 


18.1 


-3.3 


$250 to $499 


15.1 


13.4 


1.8 


$500 to $749 


14.0 


13.6 


0.4 


$750 to $1 ,000 


13.1 


10.2 


2.9 


$1 ,000 or more 


18.7 


23.2 


-4.6 


Average Earnings per Month During the 


Entire 18-Month Period (Dollars) 


568 


614 


-45 


Total Earnings During the Entire 


18-Month Period (Dollars) 


8,831 


9,381 


-550 


Sample Size 


249 


265 





Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy 
Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The data were 

weighted to account for Interview nonresponse. Standard errors of the estimates 
account for sample weights. 

7*7*** Impact estimates are statistically significant at the .05/.025/.01 level, one-talled test. 
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Table C.5. Other Characteristics of the Current or Most Recent Job Held by Sample Members 
Who Were Employed During the 18-Month Follow-Up Period 



Outcome (Percentages)® 


Program Group 


Control Group 


Occupation 


Administrative support/clerical 


1.7 


1.7 


Sales/retall 


11.2 


14.5 


Health services 


22.5 


22.4 


Food services 


19.2 


13.9 


Cleaning services 


8.1 


8.0 


Other services 


11.4 


13.4 


Production/trade 


18.4 


15.9 


Manager/professlonal/technical 


6.9 


7.8 


Other 


0.6 


2.4 


Shift or Time of Day Worked 


Regular daytime 


60.9 


68.2 


Afternoon 


2.6 


0.5 


Regular evening 


9.6 


12.2 


Overnight 


8.4 


5.2 


Rotating shift 


9.1 


6.7 


Split shift 


1.4 


0.0 


Irregular 


4.6 


5.3 


Regular with some weekends 


0.5 


0.4 


Other 


2.9 


1.6 


Self-Employed 


7.5 


9.5 


Temporary or Seasonal Job 


22.8 


21.7 


Primary Mode of Transportation to Current/Most 
Recent Job (Percentage) 


Drives self 


61.7 


65.4 


Gets ride from famlly/friends 


22.5 


19.5 


Walks 


16.8 


13.3 


Average Commute Time to Work (One-Way) 
(Minutes) 


1 5 minutes or less 


66.5 


61.7 


16 to 30 minutes 


13.9 


18.5 


31 to 45 minutes 


9.2 


10.1 


46 to 60 minutes 


1.5 


4.0 


More than 60 minutes 


9.0 


5.6 


Average 


26.2 


26.1 


Sample Size 


193 


212 



Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy 

Research, Inc. 

Note: The data were weighted to account for Interview nonresponse. Standard errors of the 

estimates account for sample weights. 



®About one-quarter of both the program and control groups did not work during the follow-up period. 
Since these cases are not Included In this table, we do not report estimated Impacts for these 
outcomes. However, we do report statistically significant differences between the two groups: 



*r*r** significantly different from zero at the .05/.025/.01 level, one-talled test. 
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Table C.6. TANF Received by Month, Based on Administrative Data 



Outcome 


Program Group 


Control Group 


Estimated Impact 


Percent Receiving TANF, by Month After 
Random Assignment 


1 


13.3 


10.6 


2.7 


2 


15.4 


12.9 


2.6 


3 


17.7 


16.3 


1.4 


4 


15.6 


13.1 


2.5 


5 


12.4 


12.1 


0.3 


6 


11.1 


14.2 


-3.1 


7 


11.2 


13.2 


-2.0 


8 


11.3 


12.8 


-1.6 


9 


10.0 


11.2 


-1.2 


10 


10.5 


8.6 


1.9 


11 


9.7 


8.4 


1.3 


12 


9.5 


7.0 


2.5 


13 


10.1 


8.9 


1.1 


14 


9.6 


8.4 


1.2 


15 


9.4 


9.1 


0.3 


16 


9.3 


9.8 


-0.5 


17 


6.8 


9.7 


-2.9 


18 


7.1 


8.8 


-1.8 


19 


7.6 


10.0 


-2.4 


20 


7.6 


9.6 


-2.0 


21 


7.4 


9.8 


-2.4 


22 


7.2 


10.6 


-3.4 


23 


5.9 


11.3 


-5.4 


24 


5.6 


9.7 


-4.0 


Average Amount of TANF Received, by 
Month After Random Assignment 


1 


36.1 


28.4 


7.7 


2 


39.6 


33.7 


6.0 


3 


46.4 


44.4 


2.0 


4 


38.5 


33.2 


5.2 


5 


31.1 


28.2 


2.8 


6 


26.9 


33.9 


-7.0 


7 


29.3 


30.6 


-1.3 


8 


32.7 


29.0 


3.7 


9 


26.3 


28.7 


-2.5 


10 


29.2 


21.2 


8.0 


11 


27.3 


19.5 


7.8 


12 


28.0 


17.6 


10.5 


13 


30.9 


24.0 


6.9 


14 


30.1 


23.0 


7.0 


15 


29.1 


24.1 


5.1 


16 


29.8 


23.9 


6.0 


17 


22.2 


24.3 


-2.2 


18 


21.7 


20.4 


1.3 


19 


22.3 


24.1 


-1.8 


20 


21.8 


21.9 


-0.2 


21 


16.7 


26.1 


-9.4 


22 


20.9 


27.4 


-6.5 


23 


17.3 


30.2 


-12.9 


24 


17.2 


27.4 


-10.2 


Sample Size 


313 


317 





Source: Administrative records data from the state of Illinois (available for 24 months after random 

assignment), compiled by Mathematica Policy Research, Inc. as part of the Rural Welfare-to- 
Work Evaluation. 

Note: All estimates were adjusted using multivariate regression methods. 

»/*./.** Impact estimates are statistically significant at the .05/.025/.01 level, one-talled test. 
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Table C.7. Food Stamps Received by Month, Based on Administrative Data 



Outcome 


Program Group 


Control Group 


Estimated Impact 


Percent Receiving Food Stamps, by Month 
After Random Assignment 


1 


61.7 


61.5 


0.2 


2 


77.4 


76.9 


0.6 


3 


76.7 


74.8 


1.9 


4 


70.5 


75.2 


-4.7 


5 


69.6 


73.2 


-3.5 


6 


66.1 


69.7 


-3.7 


7 


69.1 


66.5 


2.6 


8 


63.2 


66.0 


-2.9 


9 


64.8 


62.9 


1.9 


10 


61.8 


62.0 


-0.1 


11 


61.6 


61.8 


-0.2 


12 


61.5 


58.3 


3.2 


13 


57.8 


60.0 


-2.2 


14 


60.4 


58.9 


1.5 


15 


59.1 


55.3 


3.8 


16 


60.9 


56.3 


4.6 


17 


57.5 


55.5 


2.1 


18 


56.2 


56.5 


-0.2 


19 


57.5 


56.8 


0.7 


20 


56.4 


56.6 


-0.2 


21 


57.0 


54.5 


2.5 


22 


55.5 


57.1 


-1.6 


23 


55.7 


57.9 


-2.1 


24 


55.6 


57.1 


-1.4 


Average Amount of Food Stamps Received, 
by Month After Random Assignment 


1 


171.2 


163.7 


7.6 


2 


206.6 


208.2 


-1.6 


3 


201.3 


192.5 


8.7 


4 


187.6 


188.8 


-1.1 


5 


185.8 


189.5 


-3.7 


6 


171.2 


181.9 


-10.7 


7 


188.5 


165.5 


22.9* 


8 


175.5 


169.5 


6.0 


9 


182.3 


163.2 


19.0 


10 


168.2 


162.6 


5.6 


11 


168.7 


163.6 


5.1 


12 


173.1 


153.3 


19.8 


13 


166.8 


158.9 


7.9 


14 


177.5 


161.0 


16.5 


15 


170.2 


149.8 


20.4 


16 


176.2 


157.2 


18.9 


17 


165.4 


149.6 


15.7 


18 


160.7 


150.5 


10.2 


19 


156.9 


155.5 


1.3 


20 


156.4 


150.9 


5.4 


21 


150.9 


147.0 


3.9 


22 


152.8 


154.7 


-1.9 


23 


157.2 


154.6 


2.5 


24 


157.0 


155.1 


1.9 


Sample Size 


313 


317 





Source: Administrative records data from the state of Illinois (available for 24 months after random 

assignment), compiled by Mathematica Policy Research, Inc. as part of the Rural Welfare-to- 
Work Evaluation. 

Note: All estimates were adjusted using multivariate regression methods. 

»/*./.** Impact estimates are statistically significant at the .05/.025/.01 level, one-talled test. 
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Table C.8. Overall TANF and Food Stamp Receipt During the Foliow-Up Period, Based on Administrative 
Data 



Outcome 


Program Group 


Control Group 


Estimated Impact 


Whether Received TANF (Percentage) 


Year 1 followup 


31.0 


25.5 


5.5 


18-month followup 


32.0 


28.0 


4.0 


Year 2 followup 


15.6 


16.2 


-0.6 


Year 1 or 2 


33.8 


29.7 


4.1 


Percentage of Months Received TANF 


Year 1 followup 


12.5 


11.5 


1.0 


18-month followup 


11.4 


10.1 


1.3 


Year 2 followup 


8.0 


9.4 


-1.5 


Year 1 or 2 


10.2 


10.5 


-0.2 


Amount of TANF Received (Average) 


Year 1 followup 


391 


348 


43 


18-month followup 


555 


488 


67 


Year 2 followup 


280 


297 


-17 


Year 1 or 2 


671 


645 


26 


Whether Received Food Stamps (Percentage) 


Year 1 followup 


95.3 


92.4 


2.9 


18-month followup 


96.0 


92.5 


3.5 


Year 2 followup 


77.1 


73.1 


4.0 


Year 1 or 2 


96.6 


93.9 


2.6 


Percentage of Months Received Food Stamps 


Year 1 followup 


66.9 


67.4 


-0.5 


18-month followup 


63.9 


64.7 


-0.8 


Year 2 followup 


57.5 


56.8 


0.7 


Year 1 or 2 


62.2 


62.1 


0.1 


Amount of Food Stamps Received (Average) 


Year 1 followup 


2,180 


2,102 


78 


18-month followup 


3,197 


3,029 


167 


Year 2 followup 


1,948 


1,845 


103 


Year 1 or 2 


4,128 


3,947 


181 


Sample Size 


313 


317 





Source: Administrative records data from the state of lilinois (avaiiabie for 24 months after random 

assignment), compiied by Mathematica Poiicy Research, Inc. as part of the Rurai Weifare-to- 
Work Evaiuation. 

Note: Ali estimates were adjusted using muitivariate regression methods. 

*/**!*** Impact estimates are statistically significant at the .05/.025/.01 level, one-tailed test. 
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Table C.9. Sample Members with Income from Earnings and Other Private Sources During 
the Month Prior to the 18-Month Survey^ 



Outcome (Percentage of Sample Members) 


Program 

Group 


Control 

Group 


Estimated 

Impact 


Own Earnings 


47.7 


53.3 


-5.5 


Spouse or Partner Earnings 


13.6 


12.0 


1.6 


Other Adult Household Member Earnings 


15.9 


20.3 


-4.4 


Earnings from Informal or Odd Jobs‘s 


7.3 


7.0 


0.2 


Child Support Income 


8.5 


8.0 


0.5 


Other Private Income 


5.6 


6.9 


-1.3 


Sample Size 


252 


268 





Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy 

Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The data were 

weighted to account for Interview nonresponse. Standard errors of the estimates 
account for sample weights. 

®The month before the survey represented a different number of months after random assignment 
for different clients. For example, for some clients, the month before the survey represented 18 
months after random assignment. For others. It represented from 19 to 23 months after random 
assignment. 

‘^Earnings from Informal or odd jobs are those from either the sample member or a spouse/partner 
or other adult household member who was living with the sample member during the month before 
the survey. 

7*7*** Impact estimates are statistically significant at the .05/.025/.01 level, one-talled test. 
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Table C.10. Subgroup Differences in the Use of Services and Resources, by Period of Program Enrollment^ 





Enrolled in First Half 
of Demonstration 


Enrolled In Second Half 
of Demonstration 


Characteristic® 


Program 

Group 


Control 

Group 


Difference 


Program 

Group 


Control 

Group 


Difference 


Any Employment Preparatien Service 


53.4 


21.5 


31.9*** 


39.6 


34.9 


4.8 


Any Education or Training 


36.0 


24.0 


11.9** 


36.0 


38.1 


-2.0 


Life Skills Training 


11.7 


6.5 


5.2 


16.8 


10.2 


6.6 


Any Health-Related Service 


28.7 


28.1 


0.6 


19.8 


21.9 


-2.1 


Mediatien 


10.3 


1.4 


8.9*** 


3.7 


2.3 


1.4 


Help Paying for Child Care, 
Transportation, or Job-Related Supplies 


68.2 


36.1 


32.0*** 


51.5 


37.2 


14.3*** 


Sample Size 


137 


146 




115 


122 





Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Pellcy Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The data were weighted to account for 

interview nonresponse. Standard errors of the estimates account for sample weights. 

®The program was implemented during the two-year period from July 2001 to September 2003, and random assignment 
and program enrollment were conducted during the 18-month period from July 2001 to December 2002. Clients who 
enrolled during the first and second halves of the demonstration were those who were randomly assigned during the 
first and second 9 months, respectively, of the 18-month enrollment period. The group of clients who enrolled during 
the first half of the demenstration (July 2002 to March 2002) were mostly served during the first pregram year. 
Similarly, the group of clients who enrolled during the second half of the demonstration (April 2002 to December 2002) 
were mostly served during the second program year. 

7*7*** Differences are statistically significant at the .05/.025/.01 level, one-talled test. 
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Table C.11. Subgroup Impacts on Employment, Earnings, and Public Assistance, by Period of Program Enrollment^ 





Enrolled In First Flalf 


Enrolled In Second Flalf 




of Demonstration 


of Demonstration 






Program 


Control 


Impact 


Program 


Control 


Impact 


Outcome 


Group 


Group 


Estimate 


Group 


Group 


Estimate 


18-Month Follow-Up Survey Data 














Employed at 18-Month Followup (Percentage) 


56.1 


55.1 


0.9 


52.2 


54.7 


-2.5 


Ever Employed During 18-Month Followup (Percentage) 


77.2 


77.1 


0.1 


71.6 


72.5 


-0.8 


Tetal Earnings During 18-Month Follow-Up Period (Dollars) 


9,577 


9,221 


356 


7,799 


9,753 


-1,954 


TANF Received Last Month (Percentage) 


16.0 


14.6 


1.5 


13.8 


10.1 


3.8 


Feed Stamps Received Last Month (Percentage) 


75.2 


71.2 


4.0 


74.2 


73.8 


0.4 


Living Below Poverty at 18-Month Followup (Percentage) 


66.0 


55.8 


10.1 


70.6 


69.1 


1.5 


Sample Size 


137 


146 




115 


122 




Administrative Records Data’’ 














Ever Empleyed During 12-Month 
Follow-Up Period (Percentage) 


72.4 


67.4 


5.0 


58.2 


61.4 


-3.2 


Tetal Earnings During 12-Month Follow-Up Period (Dollars) 


3,155 


2,769 


386 


2,717 


3,144 


-426 


TANF Receipt 














Received TANF during 18-month follow-up period 
(percentage) 

Percentage of months received TANF during 18-menth 


40.3 


36.5 


3.8 


23.7 


16.0 


7.7 


follow-up period 


13.3 


13.1 


0.2 


11.0 


6.5 


4.5 


Food Stamp Receipt 














Received toed stamps during 18-month follow-up period 
(percentage) 

Percentage of months received feed stamps during 


97.6 


89.4 


8.2 


94.4 


95.7 


-1.3 


18-month follow-up period 


64.5 


63.6 


0.9 


62.3 


67.4 


-5.1 


Sample Size 


174 


175 




139 


142 
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Table C.1 1 (continued) 

Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy Research, Inc.; and administrative records 

data from the state of Illinois, compiled by Mathematica Policy Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The survey data were weighted to account for interview 

nonresponse. Standard errors of the estimates account for sample weights. 

®The program was implemented during the two-year period from July 2001 to September 2003, and random assignment and program 
enrollment were conducted during the 18-month period from July 2001 to December 2002. Clients who enrolled during the first and second 
halves of the demonstration were those who were randomly assigned during the first and second 9 months, respectively, of the 18-month 
enrollment period. The group of clients who enrolled during the first half of the demonstration (July 2002 to March 2002) were mostly served 
during the first program year. Similarly, the group of clients who enrolled during the second half of the demonstration (April 2002 to 
December 2002) were mostly served during the second program year. 

‘’We present employment data using administrative records for only 12 months (or four quarters). The available data for quarters 5 and 6 
were incomplete at the time of our analyses. In contrast, we present TANF and food stamp data for 18 months. 

7*7*** Impact estimates are statistically significant at the .05/.025/.01 level, one-tailed test. 
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Table C.12. Subgroup Differences in the Use of Services and Resources, by Clients’ Level of Employability 



Characteristic 


Less Employable 
Sample Members 


More Employable 
Sample Members 


Program 

Group 


Control 

Group 


Difference 


Pregram 

Greup 


Control 

Group 


Difference 


Any Employment Preparatien Service 


46.6 


29.3 


1 7.4*** 


47.9 


23.5 


24.3*** 


Job search/job placement assistance 


34.1 


21.8 


12.3** 


39.8 


16.6 


23.2*** 


Job readiness training 


25.2 


17.4 


7.8* 


15.6 


11.4 


4.2 


Werk-related counseling 


11.8 


4.2 


7.6** 


8.4 


2.5 


5.9* 


Any Education or Training 


36.9 


27.0 


9.9* 


37.6 


32.3 


5.4 


Life Skills Training 


13.1 


8.9 


4.2 


12.4 


7.2 


5.2 


Any Health-Related Service 


32.2 


30.1 


2.0 


14.7 


18.0 


-3.3 


Mediatien 


7.4 


2.3 


5.1* 


8.5 


0.8 


7.7** 


Help Paying fer Child Care, 














Transportation, or Job-Related Supplies 


60.1 


38.6 


21.6*** 


60.1 


33.4 


26.6*** 


Sample Size 


138 


163 




104 


97 





Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The data were weighted to account for 

interview nenresponse. Standard errors of the estimates account for sample weights. 

7*7*** Differences are statistically significant at the .05/.025/.01 level, ene-talled test. 
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Table C.13. Subgroup Impacts on Employment, Earnings, and Public Assistance, by Level of Employability 

Less Employable More Employable 

Sample Members Sample Members 





Program 


Control 


Impact 


Program 


Control 


Impact 


Outcome 


Group 


Group 


Estimate 


Group 


Group 


Estimate 


18-Month Follow-Up Survey Data 














Employed at 18-Month Followup (Percentage) 


51.2 


51.8 


-0.5 


56.8 


59.9 


-3.1 


Ever Employed During 18-Month Followup (Percentage) 


70.4 


71.3 


-0.9 


78.6 


83.2 


-4.6 


Total Earnings During 18-Month Follow-Up Period (Dollars) 


7,775 


8,150 


-375 


10,350 


11,668 


-1,318 


TANF Received Last Month (Percentage) 


16.1 


16.1 


-0.1 


7.0 


11.5 


-4.5 


Food Stamps Received Last Month (Percentage) 


77.5 


72.0 


5.5 


73.8 


67.7 


6.1 


Living Below Poverty at 18-Month Followup (Percentage) 


71.6 


67.7 


6.9 


61.6 


55.2 


6.4 


Sample Size 


138 


163 




104 


97 




Administrative Records Data’’ 














Ever Employed During 12-Month 
Follow-Up Period (Percentage) 


64.3 


63.6 


0.7 


69.7 


68.4 


1.3 


Total Earnings During 12-Month Follow-Up Period (Dollars) 


2,442 


3,069 


-627 


3,617 


2,927 


690 


TANF Receipt 














Received TANF during 18-month follow-up period 
(percentage) 

Percentage of months received TANF during 18-month 


42.3 


29.0 


13.3 


17.9 


26.1 


-8.1* 


follow-up period 


16.2 


10.7 


5.5 


6.0 


10.8 


-4.7** 


Food Stamp Receipt 














Received food stamps during 18-month follow-up period 
(percentage) 

Percentage of months received food stamps during 


97.0 


92.3 


4.1 


94.6 


90.5 


4.1 


18-month follow-up period 


66.0 


66.0 


0.1 


63.8 


60.3 


3.5 


Sample Size 


174 


175 




129 


117 
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Table C.13 (continued) 

Source: Rural Welfare-to-Work 18-Month Follow-Up Survey, conducted by Mathematica Policy Research, Inc.; and administrative records 

data from the state of Illinois, compiled by Mathematica Policy Research, Inc. 

Note: All estimates were adjusted using multivariate regression methods. The survey data were weighted to account for interview 

nonresponse. Standard errors of the estimates account for sample weights. 

®We present employment data using administrative records for only 12 months (or four quarters). The available data for quarters 5 and 6 
were incomplete at the time of our analyses. In contrast, we present TANF and food stamp data for 18 months. 

7*7*** Significantly different from zero at the .05/.025/.01 level, one-talled test. 
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